Evaluating the Accuracy of Single Camera Calibration

This example shows how to evaluate the accuracy of camera parameters estimated using the Using the Single Camera Calibrator App app or the estimateCameraParameters function.

Overview

Camera calibration is the process of estimating parameters of the camera using images of a special calibration pattern. The parameters include camera intrinsics, distortion coefficients, and camera extrinsics. Once you calibrate a camera, there are several ways to evaluate the accuracy of the estimated parameters:

• Plot the relative locations of the camera and the calibration pattern

• Calculate the reprojection errors

• Calculate the parameter estimation errors

Calibrate the Camera

Estimate camera parameters using a set of images of a checkerboard calibration pattern.

% Create a set of calibration images.
images = imageDatastore(fullfile(toolboxdir("vision"), "visiondata", ...
    "calibration", "mono"));
imageFileNames = images.Files;

% Detect calibration pattern.
[imagePoints, boardSize] = detectCheckerboardPoints(imageFileNames);

% Generate world coordinates of the corners of the squares.
squareSize = 29; % millimeters
worldPoints = patternWorldPoints("checkerboard", boardSize, squareSize);

% Calibrate the camera.
I = readimage(images, 1); 
imageSize = [size(I, 1), size(I, 2)];
[params, ~, estimationErrors] = estimateCameraParameters(imagePoints, worldPoints, ...
                                     ImageSize=imageSize);

Extrinsics

You can quickly discover obvious errors in your calibration by plotting relative locations of the camera and the calibration pattern. Use the showExtrinsics function to either plot the locations of the calibration pattern in the camera's coordinate system, or the locations of the camera in the pattern's coordinate system. Look for obvious problems, such as the pattern being behind the camera, or the camera being behind the pattern. Also check if a pattern is too far or too close to the camera.

figure; 
showExtrinsics(params, "CameraCentric");

figure; 
showExtrinsics(params, "PatternCentric");

Reprojection Errors

Reprojection errors provide a qualitative measure of accuracy. A reprojection error is the distance between a pattern keypoint detected in a calibration image, and a corresponding world point projected into the same image. The showReprojectionErrors function provides a useful visualization of the average reprojection error in each calibration image. If the overall mean reprojection error is too high, consider excluding the images with the highest error and recalibrating.

figure; 
showReprojectionErrors(params);

Estimation Errors

Estimation errors represent the uncertainty of each estimated parameter. The estimateCameraParameters function optionally returns estimationErrors output, containing the standard error corresponding to each estimated camera parameter. The returned standard error $$\sigma$$ (in the same units as the corresponding parameter) can be used to calculate confidence intervals. For example +/- $$1.96\sigma$$ corresponds to the 95% confidence interval. In other words, the probability that the actual value of a given parameter is within $$1.96\sigma$$ of its estimate is 95%.

displayErrors(estimationErrors, params);

			Standard Errors of Estimated Camera Parameters
			----------------------------------------------

Intrinsics
----------
Focal length (pixels):   [  714.1884 +/- 3.3219      710.3783 +/- 4.0579  ]
Principal point (pixels):[  563.6478 +/- 5.3967      355.7255 +/- 3.3036  ]
Radial distortion:       [   -0.3536 +/- 0.0091        0.1730 +/- 0.0488  ]

Extrinsics
----------
Rotation vectors:
                         [   -0.6096 +/- 0.0054       -0.1789 +/- 0.0073       -0.3835 +/- 0.0024  ]
                         [   -0.7283 +/- 0.0050       -0.0996 +/- 0.0072        0.1964 +/- 0.0027  ]
                         [   -0.6722 +/- 0.0051       -0.1444 +/- 0.0074       -0.1329 +/- 0.0026  ]
                         [   -0.5836 +/- 0.0056       -0.2901 +/- 0.0074       -0.5622 +/- 0.0025  ]
                         [   -0.3157 +/- 0.0065       -0.1441 +/- 0.0075       -0.1067 +/- 0.0011  ]
                         [   -0.7581 +/- 0.0052        0.1947 +/- 0.0072        0.4324 +/- 0.0030  ]
                         [   -0.7515 +/- 0.0051        0.0767 +/- 0.0072        0.2070 +/- 0.0029  ]
                         [   -0.6223 +/- 0.0053        0.0231 +/- 0.0073        0.3663 +/- 0.0024  ]
                         [    0.3443 +/- 0.0063       -0.2226 +/- 0.0073       -0.0437 +/- 0.0014  ]

Translation vectors (mm):
                         [ -146.0514 +/- 6.0391      -26.8688 +/- 3.7318      797.9024 +/- 3.9002  ]
                         [ -209.4354 +/- 6.9637      -59.4568 +/- 4.3578      921.8196 +/- 4.6295  ]
                         [ -129.3821 +/- 7.0907      -44.1033 +/- 4.3751      937.6829 +/- 4.4913  ]
                         [ -151.0046 +/- 6.6905      -27.3256 +/- 4.1339      884.2787 +/- 4.3925  ]
                         [ -174.9496 +/- 6.7056      -24.3502 +/- 4.1606      886.4959 +/- 4.6686  ]
                         [ -134.3092 +/- 7.8887     -103.4984 +/- 4.8925     1042.4551 +/- 4.8184  ]
                         [ -173.9842 +/- 7.6891      -73.1694 +/- 4.7812     1017.2383 +/- 4.8126  ]
                         [ -202.9443 +/- 7.4327      -87.9094 +/- 4.6482      983.6955 +/- 4.9072  ]
                         [ -319.8858 +/- 6.3213     -119.8901 +/- 4.0922      829.4580 +/- 4.9591  ]

How to Improve Calibration Accuracy

Whether or not a particular reprojection or estimation error is acceptable depends on the precision requirements of your particular application. However, if you have determined that your calibration accuracy is unacceptable, there are several ways to improve it:

• Modify calibration settings. Try using 3 radial distortion coefficients, estimating tangential distortion, or the skew.

• Take more calibration images. The pattern in the images must be in different 3D orientations, and it should be positioned such that you have keypoints in all parts of the field of view. In particular, it is very important to have keypoints close to the edges and the corners of the image in order to get a better estimate of the distortion coefficients.

• Exclude images that have high reprojection errors and re-calibrate.

Summary

This example showed how to obtain and interpret camera calibration errors.

References

[1] Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.