Calibration and Sensor Fusion

Interactively perform calibration, estimate lidar-camera transform, and fuse data from each sensor

Most modern autonomous or semi-autonomous vehicles are equipped with sensor suites that contain multiple sensors. It is necessary to develop a geometric correspondence between these sensors, to understand and correlate output data. Rotational and translational transformations are required to calibrate and fuse data from these sensors. Fusing lidar data with corresponding camera data is particularly useful in the perception pipeline. The lidar and camera calibration (LCC) workflow serves this purpose. It uses the checkerboard pattern calibration method. To learn more, see What Is Lidar Camera Calibration?.

Lidar Toolbox™ algorithms provide functionalities to extract checkerboard features from images and point clouds and use them to estimate the transformation between camera and lidar sensor. The toolbox also provides downstream LCC functionalities, projecting lidar points on images, fusing color information in lidar point clouds, and transferring bounding boxes from camera data to lidar data. All of these functionalities have been integrated into the Lidar Camera Calibrator app. Using the app, you can interactively calibrate the sensors.

Apps

Lidar Camera Calibrator

Interactively estimate rigid transformation between lidar sensor and camera

Functions

expand all

Feature Extraction

`estimateCheckerboardCorners3d`	Estimate world frame coordinates of checkerboard corner points in image
`detectRectangularPlanePoints`	Detect rectangular plane of specified dimensions in point cloud

Calibration

estimateLidarCameraTransform Estimate rigid transformation from lidar sensor to camera

Sensor Fusion

`projectLidarPointsOnImage`	Project lidar point cloud data onto image coordinate frame
`fuseCameraToLidar`	Fuse image information to lidar point cloud
`bboxCameraToLidar`	Estimate 3-D bounding boxes in point cloud from 2-D bounding boxes in image
`bboxLidarToCamera`	Estimate 2-D bounding box in camera frame using 3-D bounding box in lidar frame

Topics

What Is Lidar Camera Calibration?

Integrate lidar and camera data.

Calibration Guidelines and Procedure

These guidelines and procedures apply to lidar-camera calibration.

Get Started with Lidar Camera Calibrator

Interactively calibrate lidar and camera sensors.

Read Lidar and Camera Data from Rosbag File

This example shows how to read and save images and point cloud data from a rosbag file.

Featured Examples

Lidar and Camera Calibration

Estimate the rigid transformation between a 3-D lidar and a camera. At the end of this example, you will be able to use the rigid transformation matrix to fuse lidar and camera data.

Open Live Script

Multi-Lidar Calibration

Calibrate multiple 3-D lidar sensors mounted on a vehicle to estimate a relative transformation between them. Traditional methods, such as marker-based registration, are difficult when the lidar sensors have a negligible overlap between their fields of view (FOVs). The calibration also becomes more difficult as the number of lidar sensors increases. This example demonstrates the use of the trajectories of individual lidar sensors to estimate the transformation between them. This method of calibration is also known as hand-eye calibration.

Open Live Script

Detect Vehicles in Lidar Using Image Labels

Detect vehicles in lidar using label data from a co-located camera with known lidar-to-camera calibration parameters. Use this workflow in MATLAB® to estimate 3-D oriented bounding boxes in lidar based on 2-D bounding boxes in the corresponding image. You will also see how to automatically generate ground truth as a distance for 2-D bounding boxes in a camera image using lidar data. This figure provides an overview of the process.

Open Live Script

Track-Level Fusion of Radar and Lidar Data

Generate an object-level track list from measurements of a radar and a lidar sensor and further fuse them using a track-level fusion scheme. You process the radar measurements using an extended object tracker and the lidar measurements using a joint probabilistic data association (JPDA) tracker. You further fuse these tracks using a track-level fusion scheme. The schematic of the workflow is shown below.

Open Script