Converting 3D camera coordinates to robot coordinates

I have a stereovision system that is being used to a) generate 3D coordinates in terms of the camera, and b) convert these coordinates to robot coordinates. It seems that I'm having no luck with b), and I'm not sure where I'm going wrong.

I started with a 'calibration' to find the transformation matrix A. Let's say I have a point (x,y,z) in camera coords (triangulated using parameters generated in the 'Stereo camera calibrator' app). I know it is represented in the robot coords as (X,Y,Z).

I thought I would first translate the points by finding the base of the robot (0,0,0 in robot coords) in terms of the camera coordinates. Then I could do a vector subtraction to translate the point to the robot coords. Then I used y = A*x to find the transformation matrix.

This seems to work fine for the calibrated point. However, if I try to apply this transformation to another point, I get results that aren't ridiculous, but also aren't right - say 200 mm off in some directions. My errors certainly aren't that large. Is this because I'm approaching this problem wrong, or perhaps my code? I've attached my file to show what I'm doing.

%Undistort points
undistortedImPoints1 = undistortPoints(matchingPoints1,stereoParams.CameraParameters1);
undistortedImPoints2 = undistortPoints(matchingPoints2,stereoParams.CameraParameters2);

% Triangulate the Undistorted Image Points
[calculatedPoints1,errors] = triangulate(undistortedImPoints1,undistortedImPoints2,stereoParams);

base1 = [468,238];
base2 = [621,325];
%Undistort points
undistortedImPointsbase1 = undistortPoints(base1,stereoParams.CameraParameters1);
undistortedImPointsbase2 = undistortPoints(base2,stereoParams.CameraParameters2);

% Triangulate the Undistorted Image Points
[base,errors] = triangulate(undistortedImPointsbase1,undistortedImPointsbase2,stereoParams);

vectorproj = (calculatedPoints1(1,:)-base(1,:));
actualpoint = extrinsicWorldPoints(1,:);
y = actualpoint';
x = vectorproj';
A = y/x;
%Check
y = A*x;

try1 = [153,223];
try2 = [225,380];
undistortedImPoints1 = undistortPoints(try1,stereoParams.CameraParameters1);
undistortedImPoints2 = undistortPoints(try2,stereoParams.CameraParameters2);
[trying,errors] = triangulate(undistortedImPoints1,undistortedImPoints2,stereoParams);
vectorproj = (trying-base(1,:))'
tryresult = (A*vectorproj)'