syms a b c d ;
b=c+j*d;
Okay, b is a symbolic number that might or might not be complex. "might not be" because d could be 0, or because d could be imaginary-valued -- you did not tell MATLAB that c or d are confined to real values, so MATLAB cannot tell that b is complex.
fun3=@(a,b)2*a^2+6*(a+b);
The a and b in the body of fun3 have nothing to do with the syms a b or b = c+j*d .
The rule for anonymous functions is that the body of the function is gone through, and any variable that appears inside the @() is replaced with an internal reference to the corresponding positional parameter at runtime. Any variable that appears in side the body that is not listed inside @() is searched for in the current workspace (and possibly nesting workspaces), and if it is found then a copy of the value of the variable as of the time the @() is executed is copied into memory associated with the function handle. In the expression @(a,b)2*a^2+6*(a+b) then all of the variables that appear are inside the @() so nothing is copied from memory; the expression is just turned into references to positional parameters, like
@(varargin) 2*varargin{1}^2 + 6*(varargin{1}+varargin{2})
To repeat myself, the b you name in the @(a,b) will have nothing to do with the b=c+j*d . The @(a,b) temporarily over-shadows the meaning of a and b inside the body of the function being defined.
So now fun3 is a function that accepts two arguments and does a calculation, and has nothing to do with b=c+j*d
Then fun2 is defined as accepting a vector with at least two values and separating the values out and passing them to the fun3 function handle. The effect is much like if you had coded
fun2 = @(varargin) 2*varargin{1}(1)^2 + 6*(varargin{1}(1)+varargin{1}(2))
After that, fun2 is passed to fmincon, along with a purely real-valued x0. When you have real-valued x values, the 2*x1^2 + 6*(x1+x2) process is going to produce a real output, so when you start with real x0, fmincon will never be able to escape to complex values. You aren't solving for roots, you are minimizing, and the minimum of a real-valued expression (given real-valued starting points) is whatever it is.
If you want b to be complex valued, then for the purposes of fmincon it is much easier to make the real and imaginary portion into separate variables:
fun3 = @(x) 2*x(1)^2 + 6 * (x(1) + x(2) + 1j*x(3))
Now you encounter the problem that this would typically give you a complex result, and fmincon is a minimizer, and minimizing a complex number is not defined, You can try to minimize the norm of the fun3:
fun2 = @(x) norm(fun3(x))
With a small bit of thought, we can read off the solution: the norm would be minimized at a = 0, b = 0 + 0*1j which would be norm 0.
