Matlab Stereo Camera Calibrator Scene Reconstruction Error

I am trying to use the Computer Vision System Toolbox to calibrate the pair of cameras below in order to be able to generate a 3-D point cloud of a vehicle at a range between 1 to 5m. The output image size was approximately 1 MB per image for the checkerboard calibration images and the checkerboard square size was 25 mm. The cameras used were a pair of SJ4000 HD1080P cameras. The cameras were placed as parallel to each other as possible with no angle in the vertical axis. The checkboard calibration was done with the aid of a bright light and whiteboard. The mean error per pixel using the stereo camera calibrator code was 3.31 with 31/32 successful pairings. The approximate distance to the checkerboard was 30 cm and the distance between the cameras was 20 cm.

The problem I am encountering upon rectification is during the 3D Reconstruction of the scene. The figure below is what was outputted. I’m not sure if there is a parameter that is missing in the camera setup or if there is something that is missing / needs to be added in the script. Below is the code that was used for the Stereo anaglyph and scene reconstruction that was adapted from the Matlab Stereo Camera Calibration tutorial.

% Read in the stereo pair of images.
I1 = imread('left.jpg');
I2 = imread('right.jpg');

% Rectify the images.
[J1, J2] = rectifyStereoImages(I1, I2, stereoParams);

% Display the images before rectification.
figure;
imshow(stereoAnaglyph(I1, I2), 'InitialMagnification', 50);
title('Before Rectification');

% Display the images after rectification.
figure;
imshow(stereoAnaglyph(J1, J2), 'InitialMagnification', 50);
title('After Rectification');
% 
% Compute Disparity for 3-D Reconstruction 
% The distance in pixels between corresponding points in the rectified images is called disparity. 
% The disparity is used for 3-D reconstruction, because it is proportional to the distance between the cameras and the 3-D world point.
disparityMap = disparity(rgb2gray(J1), rgb2gray(J2));
figure;
imshow(disparityMap, [0, 64], 'InitialMagnification', 50);
colormap('jet');
colorbar;
title('Disparity Map');

%Reconstruct the 3-D Scene
%Reconstruct the 3-D world coordinates of points corresponding to each pixel from the disparity map.

point3D = reconstructScene(disparityMap, stereoParams);

% Convert from millimeters to meters.
point3D = point3D / 1000;

% Visualize the 3-D Scene
% Plot points between 3 and 7 meters away from the camera.
z = point3D(:, :, 3);
zdisp = z;
point3Ddisp = point3D;
point3Ddisp(:,:,3) = zdisp;
showPointCloud(point3Ddisp, J1, 'VerticalAxis', 'Y',...
    'VerticalAxisDir', 'Down' );
xlabel('X');
ylabel('Y');
zlabel('Z');

I have included the images of the Scene Reconstruction, Disparity Map, Mean Error Per Pixel and After Rectification. The version of Matlab I am using is R2014b Student Edition purchased from the Matlab Website.

Thank you very much for any help

Answer 1

[Dima Lisin]

You have two problems here. One is that the lens distortion may simply be too severe. The best thing to try here is to take even more calibration images so that you have the checkerboard close to the edges and corners of the field of view. Also, try placing the checkerboard at varying distances from the camera. See if you can get those reprojection errors down.

The second issue here is that you need to change the 'DisparityRange' parameter of the disparity function. Display the anaglyph image using imtool, and use the ruler widget to measure distances between some pairs of corresponding points. That should give you the idea of what the disparity range should be. Just looking at the image I can see that [0 64] is way too small.

Answer 2

[Munin]

There is additional custom logic required to restrict the point cloud output from 1600m to the 3-4m in that scene:

https://github.com/SNDST00M/stereo2stl/blob/v1.5.0/script.m#L216-L380