Don't try to call fzero with a function handle to the function containing the fzero call as the input. If you do, the function will call fzero which will call the function which will call fzero which will call the function which will call fzero ... until eventually you reach the recursion limit and MATLAB throws an error or you exhaust the stack space available to MATLAB and MATLAB and/or your computer crashes.
Call fzero from a different function than the one you're trying to solve. I'm going to return the function handle from the "different function" so you can check the answer. I also had to change the interval, since your function was positive for both x = 0.5 and x = 4 which caused fzero to throw an error. Plotting your function shows a zero between x = 1 and x = 2 (and the sign of the function value is different for those two points.)
function [x, fun] = example542639
fun = @f; % function
x0 = [1 2]; % initial interval
x = fzero(fun,x0);
function y=f(x)
y=x.^3-2*x.^2*sin(x)+5*x*cos(x)+1/x;
How to use this?
[x, fun] = example543639; % call the function
result = fun(x) % Check that x is a zero of fun, result should be small in magnitude
