Please let us know what error exactly you are getting while using Function block in Simulink.
My Initial guess is that error might be coming because iav is returning as empty symbolic variable . I tried to run the your function on your end and the value you passing to the function is not getting assigned to the symbolic Variable xi.
Also it is advised to convert the iav to double before returning.
function t = fcn(txi)
syms xi phi iav
xi = txi;
fun = 1/(sqrt(1-(1-phi^(3/2))*((1-iav)*xi)^(3/2)));
eqn = (1-iav)*int(fun,phi,0,1)==1;
iav = vpasolve(eqn,iav);
t = double(iav);