Lidar camera calibration helps to estimate relative position and orientation between a lidar and a camera in a system
Cameras provide rich color information, while lidar sensors provide an accurate 3-D structural and locational information of objects. When fused together, we can enhance the performance of perception and mapping algorithms for autonomous driving and robotics applications.
Lidar camera calibration involves calculation of extrinsic parameters of the lidar-camera system in the form of a rigid transformation matrix.
Extrinsic parameters define the location and orientation of the sensors with respect to the world frame and with each other. These parameters are essential to fuse data from these sensors. Lidar Toolbox provides both MATLAB functions and an interactive app for performing lidar- camera calibration. The Lidar Camera Calibrator app is an easy and interactive interface to cross calibrate cameras and lidars in a system.
Here is the workflow we will be following. We will load and then extract checkerboard features from images and corresponding point clouds. We will then use these features to estimate transformation matrix between camera and lidar.
Let’s now see how to use Lidar Camera Calibrator app to implement this workflow.
We can open the app using command line or you find the app from add on menu.
Once the app is open, we will first add images and the corresponding point clouds of a checkerboard calibration pattern. A checkerboard is used because its regular pattern makes it easy to detect automatically. It is recommended to use between 10 and 20 checkerboard data for accurate calibration results.
Next, enter the size of the checkerboard square in world units in millimeters, centimeters, on in meters. Here for this data, we are using a checkerboard square of 200 mm in size.
This step is necessary to find the mapping between world units with the image-point cloud pairs.
While we load the data, the app automatically calibrate the camera intrinsics and detects the checkerboard calibration pattern in the loaded image-point cloud pairs.
Alternatively, if you have camera intrinsic already available, you can load them by choosing use fixed intrinsics in camera intrinsics section.
In the left panel you can see accepted data and rejected data. Every image-point cloud pair from which checkerboard pattern was detected can be seen in the accepted data panel. These data will be used to calibrate the transformation between the sensors.
You can tweak the values on ‘Feature Detection’ section on toolstrip to help with detection in rejected data also
You can also use the edit ROI functionality to improve your detection. You can change/ move ROI to capture the region of point cloud that consist of the checkerboard.
Then click ‘Detect’ to detect checkerboard patterns on the entire data again. You can see here that 8 out of 9 pairs are now accepted for calibrating the transformation.
After checkerboard detection, we can calibrate the transformation by clicking ‘calibrate’ button.
You can now see lidar checkerboard points projected on image, and image colors fused into lidar point cloud.
Once calibration is done, you can evaluate calibration results by visualizing translation, rotation, and reprojection errors.
This is helpful to identify bad data that you can remove and recalibrate for better results.
We can do this by dragging the threshold line in any of the error plots and then deleting the highlighted data pairs in accepted data browser by right clicking and selecting ‘remove and recalibrate’ option.
Now we can export the estimated transformation and errors to the MATLAB workspace, or to a file.
You can also generate a MATLAB script out of it.
A calibrated lidar camera system can be further used for applications like
1- estimating coordinates of 3D bounding boxes in lidar data from 2D bounding boxes in the corresponding image.
2- Fusing color information from camera on point cloud.
Refer to MathWorks documentation and Lidar Toolbox product pages to learn more.
If you have any questions or comments, please let us know.
