If your goal is to compute radii of curvature at some maximum point on the surface, Sean suggests a sphere fit, which is I think in the right direction, but not the right answer.
I'd suggest that you choose a set of points that lie in the immediate vicinity of your chosen location. Then run an ellipsoid fit through them.
An ellipsoid will have three axes of interest here.
However, there is NO assurance at any arbitrary location on the surface, that the surface even has the correct sign on those curvatures for any such fit to be correct. For example, suppose you choose a saddle point? Here we might have a positive curvature in one arbitrary direction, and negative one in the perpendicular direction. And there is no assurance at all that these directions will be aligned with the axes. So really, even an ellipsoid fit is inappropriate in general.
Perhaps a better choice is to fit a general complete quadratic polynomial model at any location to the points in the immediate vicinity. You could then trivially compute a Hessian matrix from that model. (In fact, if you have a gridded surface, you should be able to do it all with three calls to conv2, which would give you estimated hessian matrices over the entire gridded domain. This only works if you have a nice 2-d rectangularly gridded surface though, and I gather you don't have that, but a triangulated surface.)
Anyway, once you have a Hessian matrix, the radius of curvature in any direction is easily computed, regardless of the local shape.
