Getting a plane to be perpendicular to the Z axis.

I have a collection of about 500 points which lie roughly along a plane. I need to rotate the plane fit of this data to lie perpendicular to the Z axis. I can't seem get the plane any flatter than a slope of ~ 7*10^-3

Currently I am crossing the normal vector of the plane with [0,0,1] to find an axis of rotation. Then asin of the magnitude of the cross product to find the angle to rotate. Then put the angle and axis into a rotation matrix from wikipedia ( http://en.wikipedia.org/wiki/Rotation_matrix the one labeled "Rotation matrix from axis and angle") and multiply my points in.

I have tried using format long and a "trig-free" method of replacing the trig functions in the rotation matrix with their actual terms so as to avoid any error in the asin function. Both of these things made no difference.

any insight as to how to get the plane flat would be appreciated. I need it to have a slope of <10^-6.

[Walter Roberson]

What is the maximum distance of points from the origin?

[Adam]

as of right now the plane is centered on the origin with z values of +/- 7*10^-4 and x y values of +/- 5*10^-2

Answer 1

[Jan]

ASIN and ACOS are numerical instable at some locations, such that accurate measurements of angles are expected by ATAN2 only. See http://www.mathworks.de/matlabcentral/newsreader/view_thread/151925

[Adam]

getting the same results with atan2, however we got it flat doing a z rotation and then a y rotation instead of the arbitrary axis rotation. Very bizarre imo.