Ground Truth Labeler

Label ground truth data for automated driving applications

Description

The Ground Truth Labeler app enables you to label ground truth data in multiple videos, image sequences, or lidar point clouds.

Using the app, you can:

• Simultaneously label multiple time-overlapped signals representing the same scene.

• Define rectangular region of interest (ROI) labels, polyline ROI labels, pixel ROI labels, cuboid ROI labels for lidar labeling, and scene label definitions. Use these labels to interactively label your ground truth data.

• Use built-in detection or tracking algorithms to label ground truth data.

• Write, import, and use custom automation algorithms to automatically label ground truth data.

• Evaluate the performance of your label automation algorithms by using a visual summary.

• Export the ground truth labels as a groundTruthMultisignal object. You can use this object for system verification or for training an object detector or semantic segmentation network.

• Display time-synchronized signals, such as CAN bus data, by using the driving.connector.Connector API.

To learn more about this app, see Get Started with the Ground Truth Labeler.

×

Open the Ground Truth Labeler App

• MATLAB^® Toolstrip: On the Apps tab, under Automotive, click the app icon.

• MATLAB command prompt: Enter groundTruthLabeler.

Examples

Related Examples

• Get Started with the Ground Truth Labeler
• Automate Ground Truth Labeling of Lane Boundaries
• Automate Ground Truth Labeling for Semantic Segmentation
• Automate Attributes of Labeled Objects
• Evaluate Lane Boundary Detections Against Ground Truth Data
• Evaluate and Visualize Lane Boundary Detections Against Ground Truth

Programmatic Use

groundTruthLabeler

groundTruthLabeler opens a new session of the app, enabling you to label ground truth data.

groundTruthLabeler(videoFileName)

groundTruthLabeler(videoFileName) opens the app and loads the input video. The video file must have an extension supported by VideoReader.

Example: groundTruthLabeler('caltech_cordova1.avi')

groundTruthLabeler(imageSeqFolder)

groundTruthLabeler(imageSeqFolder) opens the app and loads the image sequence from the input folder. An image sequence is an ordered set of images that resembles a video.

imageSeqFolder must be a string scalar or character vector that specifies the folder containing the image files. The image files must have extensions supported by imformats and are loaded in the order returned by the dir function.

The images in imageSeqFolder must be the same size. If the images vary in size, the app imports only the images that are of the same size as the first image in the sequence. To label a collection of unordered images that vary in size, use the Image Labeler (Computer Vision Toolbox) app instead.

groundTruthLabeler(imageSeqFolder,timestamps)

groundTruthLabeler(imageSeqFolder,timestamps) opens the app and loads a sequence of images with their corresponding timestamps. timestamps must be a duration vector of the same length as the number of images in the sequence.

For example, load a sequence of road images and their corresponding timestamps into the app.

imageDir = fullfile(toolboxdir('driving'),'drivingdata','roadSequence');
load(fullfile(imageDir,'timeStamps.mat'))
groundTruthLabeler(imageDir,timeStamps)

groundTruthLabeler(___,'ConnectorTargetHandle',connector)

groundTruthLabeler(___,'ConnectorTargetHandle',connector) opens the app and loads both of these components:

• A video or image sequence signal, depending on the input argument combination you specify

• An external analysis or visualization tool that is time-synchronized with the specified signal

The connector input is a handle to a driving.connector.Connector class that implements the external tool.

For example, this syntax opens the app with a video signal and synchronized lidar visualization tool.

groundTruthLabeler('01_city_c2s_fcw_10s.mp4','ConnectorTargetHandle',@LidarDisplay);

When you have an external tool connected to a signal in the app, consider these tips.

• If you remove the signal that is connected to the tool, the app disconnects the tool and closes it.

• The signal connected to the tool must be the master signal, that is, the signal whose timestamps are used in the playback controls at the bottom of the app. If you change the master signal, the app disconnects the tool and closes it.

• If you start a new app session, the app disconnects the tool and closes it.

groundTruthLabeler(sessionFile)

groundTruthLabeler(sessionFile) opens the app and loads a saved app session, sessionFile. The sessionFile input contains the path and file name. The MAT-file that sessionFile points to contains the saved session.

Limitations

• Lidar signals do not support line or pixel ROI labels.

• Pixel ROI labels do not support sublabels or attributes.

• Cuboid ROI labels do not support sublabels.

• The Label Summary window does not support sublabels or attributes

More About

ROI Labels, Sublabels, and Attributes

On the left side of the app, the ROI Labels pane contains the region of interest (ROI) label definitions that you can mark on the frames. You can create label definitions directly from this pane. Alternatively, you can create label definitions programmatically by using a labelDefinitionCreatorMultisignal object and then import these label definitions into an app session.

The app supports the definition of ROI labels, sublabels, and attributes.

ROI Labels

An ROI label is a label that corresponds to a region of interest (ROI) in a signal frame. The table describes the supported label types.

ROI Label	Description	Example: Driving Scene
Projected cuboid	Draw cuboidal ROI labels (3-D bounding boxes).
Rectangle/Cuboid	Draw rectangular or cuboidal ROI labels around objects, depending on the signal type. In image signals, draw rectangular ROI labels (2-D bounding boxes). In lidar signals, draw cuboidal ROI labels (3-D bounding boxes). For more on lidar labeling, see Label Lidar Point Clouds for Object Detection.	Vehicles, pedestrians, road signs Rectangle: Cuboid:
Line	Draw linear ROI labels to represent lines. To draw a polyline ROI, use two or more points.	Lane boundaries, guard rails, road curbs
Pixel label	Assign labels to pixels for semantic segmentation. You can label pixels manually using polygons, brushes, or flood fill. For more on pixel labeling, see Label Pixels for Semantic Segmentation (Computer Vision Toolbox).	Vehicles, road surface, trees, pavement

ROI Sublabels

An ROI sublabel is an ROI label that belongs to a parent label. Use ROI sublabels to provide a greater level of detail about the ROIs in your labeled ground truth data. For example, a vehicle label might contain headlight, licensePlate, and wheel sublabels. You can create sublabels only for rectangular and polyline labels. For more details about sublabels, see Use Sublabels and Attributes to Label Ground Truth Data (Computer Vision Toolbox).

ROI Attributes

An ROI attribute specifies additional information about an ROI label or sublabel. For example, in a driving scene, attributes might include the type or color of a vehicle. The table describes the supported attribute types.

Attribute Type	Sample Attribute Definition	Sample Default Values
Numeric Value
String
Logical
List

For more details about attributes, see Use Sublabels and Attributes to Label Ground Truth Data (Computer Vision Toolbox).

Tips

• To avoid having to relabel ground truth with new labels, organize the labeling scheme you want to use before marking your ground truth.

• You can copy and paste labels between signals that are of the same type.

Algorithms

You can use label automation algorithms to speed up labeling within the app. To create your own label automation algorithm to use within the app, see Create Automation Algorithm for Labeling (Computer Vision Toolbox). You can also use one of the provided built-in algorithms. Follow these steps:

1. Load the data you want to label, and create at least one label definition.

2. On the app toolstrip, click Select Algorithm, and select one of the built-in automation algorithms.

3. Click Automate, and then follow the automation instructions in the right pane of the automation window.

ACF Vehicle Detector

Detect and label vehicles using aggregate channel features (ACF). This algorithm is based on the vehicleDetectorACF function. To use this algorithm, you must define at least one rectangle ROI label. You do not need to draw any ROI labels.

To help improve the algorithm results, first click Settings. You can change any of these settings.

• The pretrained vehicle detector model that the algorithm uses — The 'full-view' model was trained using unoccluded images of the front, rear, left, and right sides of vehicles. The 'front-rear-view' model was trained using images of only the front and rear sides of the vehicle.

• The overlap ratio threshold, from 0 to 1, for detecting vehicles — When rectangle ROIs overlap by more than this threshold, the algorithm discards one of the ROIs.

• The classification score threshold for detecting vehicles — Increase the score to increase the prediction confidence of the algorithm. Rectangles with scores below this threshold are discarded.

You can also configure the detector with a calibrated monocular camera by importing a monoCamera object into the MATLAB workspace. Specify the length and width ranges of the vehicle in world units, such as meters.

ACF People Detector

Detect and label people using aggregate channel features (ACF). This algorithm is based on the peopleDetectorACF (Computer Vision Toolbox) function. To use this algorithm, you must define at least one rectangle ROI label. You do not need to draw any ROI labels.

To help improve the algorithm results, first click Settings. You can change any of these settings.

• The pretrained people detector model that the algorithm uses — The 'inria-100x41' model was trained using the INRIA person data set. The 'caltech-50x21' model was trained using the Caltech Pedestrian data set.

• The overlap ratio threshold, from 0 to 1, for detecting people — When rectangle ROIs overlap by more than this threshold, the algorithm discards one of the ROIs.

• The classification score threshold for detecting people — Increase the score to increase the prediction confidence of the algorithm. Rectangles with scores below this threshold are discarded.

Point Tracker

Track and label one or more rectangle ROI labels over short intervals by using the Kanade-Lucas-Tomasi (KLT) algorithm. This algorithm is based on the vision.PointTracker (Computer Vision Toolbox) System object™. To use this algorithm, you must define at least one rectangle ROI label, but you do not need to draw any ROI labels.

To change the feature detector used to obtain the initial points for tracking, click Settings. This table shows the feature detector options.

Feature Detector	Description	Equivalent Function
Minimum Eigen Value	Detect corners by using the minimum eigenvalue algorithm.	detectMinEigenFeatures (Computer Vision Toolbox)
Harris	Detect corners by using the Harris–Stephens algorithm.	detectHarrisFeatures (Computer Vision Toolbox)
FAST	Detect corners by using the features from accelerated segment test (FAST) algorithm.	detectFASTFeatures (Computer Vision Toolbox)
BRISK	Detect features by using the binary robust invariant scalable keypoints (BRISK) algorithm.	detectBRISKFeatures (Computer Vision Toolbox)
KAZE	Detect features by using nonlinear diffusion to construct a scale space of an image, and then detecting multiscale corner features (KAZE features) from that scale space.	detectKAZEFeatures (Computer Vision Toolbox)
SURF	Detect blob features by using the speeded-up robust features (SURF) algorithm.	detectSURFFeatures (Computer Vision Toolbox)
MSER	Detect regions by using the maximally stable extremal regions (MSER) algorithm.	detectMSERFeatures (Computer Vision Toolbox)

Temporal Interpolator

Estimate rectangle ROIs between frames by interpolating the ROI locations across the time interval. To use this algorithm, you must draw a rectangle ROI on a minimum of two frames: one at the beginning of the interval and one at the end of the interval. The interpolation algorithm estimates and draws ROIs in the intermediate frames.

Consider a video with 10 frames. The first frame has a rectangle ROI centered at [5, 5]. The 10th frame has a rectangle ROI centered at [25, 25]. At each frame, the algorithm moves the ROI 2 pixels in the x-direction and 2 pixels in the y-direction. Therefore, the algorithm centers the ROI at [7, 7] in the second frame, [9, 9] in the third frame, and so on, up to [23, 23] in the second-to-last frame.

Point Cloud Temporal Interpolator

Estimate cuboid ROIs between point cloud frames by interpolating the ROI locations across the time interval. To use this algorithm, you must draw a cuboid ROI on a minimum of two frames: one at the beginning of the interval and one at the end of the interval. The interpolation algorithm estimates and draws ROIs in the intermediate frames.

Consider a point cloud sequence with 10 frames. The first frame has a cuboid ROI centered at [5, 5, 0]. The 10th frame has a cuboid ROI centered at [25, 25, 0]. At each frame, the algorithm moves the ROI 2 points in the x-direction, 2 points in the y-direction, and 0 points in the z-direction. Therefore, the algorithm centers the ROI at [7, 7, 0] in the second frame, [9, 9, 0] in the third frame, and so on, up to [23, 23, 0] in the second-to-last frame.

Lane Boundary Detector

Detect and label lane boundaries using an estimated bird’s-eye-view projected image. To use this algorithm, you must define at least one line ROI label. You do not need to draw any ROI labels. To detect lane boundaries, the algorithm follows these steps:

1. It makes an initial guess at the placement of the lane boundaries in the image.

2. It transforms the ROI around the lanes into a bird's-eye view image to make the lanes parallel and remove distortion.

3. It uses this image to segment the lane boundaries.

To help improve the algorithm results, first click Settings. You can change any of these settings.

• The placement of the lane lines for generating the bird's-eye view image

• The ROI around the lanes, which you can expand to include more than just the ego lane boundaries in the image

• The pixel width of detected lane boundaries in the image

You can also change the number of lane boundaries that you want to detect. The default number of lane boundaries is 2.

Compatibility Considerations

Ground Truth Labeler app no longer exports groundTruth objects

Behavior change in future release

If you import labels or open an app session created before R2020a, the Ground Truth Labeler exports labeled data as a groundTruthMultisignal object instead of as a groundTruth (Computer Vision Toolbox) object.

If you do not need to label multiple signals simultaneously and do not require lidar labeling, import the labels or session into the Video Labeler (Computer Vision Toolbox) app instead. The Video Labeler app continues to export groundTruth objects.