Determination of stereo camera world coordinates with respect to calibration target

I'm having some issues getting a precise and accurate determination of the location of my cameras. I am defining my world coordinate system with the origin on the calibration target, and want to know the world coordinates of the camera centers relative to this. I take a set of ~20 images, calibrate the cameras using the stereo calibration functions, and then compute the camera centers as:

wocoIdx= 1;   % index for the image with the calibration target located where I would like to define my world coordinate system
C1= -stereoParams.CameraParameters1.TranslationVectors(wocoIdx,:);
C2= -stereoParams.CameraParameters2.TranslationVectors(wocoIdx,:);

The issue is that this typically gives me cameras spaced ~111mm apart horizontally, versus the 140mm I have measured manually.

(1) Is camera height vs. camera spacing a limiting factor here? My cameras are about 1960mm from the world coordinate target and spaced ~140mm apart. When calibrating, I translate the calibration target in a range of 200mm towards/away from the camera (this is limited by the depth of field of the setup).

(2) Am I missing another step (rotation?) to obtain the camera centers? I've also tried using the triangulate function but had little success getting an accurate estimate. The approach I used with triangulate is:

undistortedImagePts1= undistortPoints(imagePoints(:,:,wocoIdx,1),stereoParams.CameraParameters1);
undistortedImagePts2= undistortPoints(imagePoints(:,:,wocoIdx,2),stereoParams.CameraParameters2);
C1= triangulate(undistortedImagePts1(1,:),undistortedImagePts2(1,:),stereoParams);
C2= C1 + stereoParams.TranslationOfCamera2';

Answer 1

[Dima Lisin]

Matt's is almost correct. The extrinsics R and t represent the transformation from the world coordinates into camera's coordinates. So t is not the camera center. You have to rotate as well. The only problem with Matt's solution is that it does not follow the vector-matrix multiplication convention used by the Computer Vision System Toolbox. The camera center in the world coordinates is

c = -t * R';

because t is a row vector.

Also, you do not have to use one of your calibration images. You can calibrate your cameras and then take a new picture of a checkerboard, and use the extrinsics function to compute the R and t relative to that board. See this example.

[Tracy]

Thanks for the clarification, Dima.

I changed that in my code, but I'm still getting rather different camera spacings depending on how far my target is from the cameras during calibration -- differences on the order of 33mm. I suppose I could calibrate with the target much closer to the cameras, and then refocus the cameras at a farther plane. This would only change the intrinsics, correct, and then I could use the existing stereoParams to call the extrinsics function for a new calibration image?

[Tracy]

Also, are there any rules of thumb about stereo camera geometry for calibration, e.g. how close the target should be to the cameras, how far apart the cameras should be spaced, etc. based on the properties of the cameras and lenses?

[Dima Lisin]

The baseline (distance between cameras) should be proportional to the distance to the object. If the object is too far away, then stereo disparity becomes too small to be measured. This is when you may want to increase the baseline. On the other hand, if the baseline is too wide, then you cannot measure the objects that are close. It is like moving your finger too close to your eyes. This also depends on the resolution of your cameras.

You can easily check if your setup is reasonable. Calibrate your cameras, take a pair of images, rectify them, create a red-cyan anaglyph from the rectified images using stereoAnaglyph function, and display it using imtool. Then use the ruler widget in imtool to measure the disparity between a few pairs of corresponding points. If the disparity is less than a pixel, than you want to increase baseline or move the cameras closer to the object. If the disparity is too large (more than a couple of hundred pixels), then reduce the baseline, or move the object farther away.

Answer 2

[Matt J]

Am I missing another step (rotation?) to obtain the camera centers?

I think so. The camera center is the null vector of the 3x4 camera matrix or, in non-homogenous coordinates, the center C is given by

C=-R.'*t

where R is the camera rotation matrix and t is the translation vector from the camera extrinsic parameters. It looks like you are interpreting t to be the camera center.

[Tracy]

Thanks for the reply. That's possible, but it doesn't seem to make a very big difference.

I've added:

R1= rodrigues(stereoParams.CameraParameters1.RotationVectors(wocoIdx,:));
R2= rodrigues(stereoParams.CameraParameters2.RotationVectors(wocoIdx,:));

and changed C1, C2 to:

 C1= -R1*stereoParams.CameraParameters1.TranslationVectors(wocoIdx,:);
 C2= -R2*stereoParams.CameraParameters2.TranslationVectors(wocoIdx,:);

The horizontal distance between the two cameras is still ~111mm. The function rodrigues I'm using comes from this file in the Caltech vision toolbox, for converting a rotation vector to a 3x3 rotation matrix using the Rodrigues formula.

[Matt J]

Did you do anything to verify the success of the calibration? When you compose the camera matrices and apply them to sets of world coordinates (e.g., the origin) do the world coordinates project onto the appropriate 2D coordinates in the camera images?

[Tracy]

Yes, I've computed the reprojection error (it's on the order of 0.1-0.2 pixels) and done some visual checks, e.g. converting pixel coordinates of a point in cam1 to world coordinates, converting these world coordinates to image coordinates of cam2, and confirming that these points match up.

[Matt J]

Hmmm. Then, I guess my next question would be how you did the manual measurement (the one which gave you 140 mm separation). In my (limited) experience, the physical location of the camera center isn't something that's very easy to know in advance. I believe it might not even necessarily be located within the physical shell of the camera.

[Tracy]

Hmmm, that's an interesting idea. I just measured the distance between cameras with a ruler. These are very small cameras (54 x 29 x 29 mm), so I was assuming that there wasn't much variation in possible location of the camera center between the two. I hadn't considered that the optical center might not physically be inside the camera/lens setup though. This paper talks a bit about finding the optical center of a camera, but (skimming) doesn't seem to go into that question.

[Matt J]

Another suggestion would be to try the calibration again with the cameras at a smaller depth from the calibration target. I've heard speculation that the decomposition of a camera into extrinsic and intrinsic parameters becomes more ill-posed at large depths. If a smaller depth gives you better accuracy, it could at least tell you whether the camera centers are located in the physical bodies of the camera.

[Tracy]

Great idea for checking. Recalibrating with the target closer to the cameras (~1.2 meters away instead of ~1.8) gives a spacing between cameras of 147mm (which is closer to what I'd expect). Tomorrow I'll try spacing the cameras farther apart for the longer depth of field, to see if it's a camera height : camera spacing issue like I suspected.

Thanks for your help, Matt! Would of course welcome any more advice but wanted to make sure I expressed my appreciation for your suggestions.