The problem lies in your code. First, you have an obvious error in the code you show. I'll guess you just used too many or too few parens in the first term for fun. I'll delete the extra right paren on this term:
as it is spurious. But now look at the expression:
fun=q*cos(phi)^2/4 + (p*sin(phi)^2)/4 + (p*cos(theta/2)^2)/8 - (p*cos(xi)^2*cos(xi)^2)/4-0.238
Next, you write this line:
which is completely meaningless in context, since gg is undefined. Sigh. If you want help, one would think you would at least test the code you show to insure it does what you claim it does This forces me to make wild guasses as to what you mean.
If looks like you may have wanted to write
So matlabFunction created a function handle, which you then go on to use in an optimization. (MAYBE that is what you did. But at this point, I'm just making wild random guesses.)
Do you see the order of the parameters in fun? matlabFunction chose them in what looks like alphabetical order, if you do not tell it to do otherwise. But you can force it to use the order you wish. For example:
And now your optimization will use these variables in the expected order.