Create Instance Segmentation Training Data from Labeled Ground Truth

R2026b

After you label object instances with polygon ROIs using the Image Labeler or Video Labeler app and export a groundTruth object, you must convert the polygon annotations into a training datastore before you can train an instance segmentation network such as SOLOv2 or Mask R-CNN. The training datastore must return data as a 1-by-4 cell array in {image, boxes, labels, masks} format. This topic shows you how to create that datastore using the instanceSegmentationTrainingData function. To get started with labeling ground truth, see Label Objects Using Polygons for Instance Segmentation.

To perform transfer learning using a SOLOv2 or Mask R-CNN network, you must train the network on a custom ground truth data set. Before training, you must first postprocess the ground truth labels exported from the Image Labeler or Video Labeler app to ensure the annotations meet the network requirements, such as correct label and mask formatting. Then, convert the postprocessed ground truth data into a datastore of the format required by the training data argument of the training function for your desired instance segmentation network.

To learn more, see Get Started with Instance Segmentation Using Deep Learning.

Create Training Datastore from Single groundTruth Object

Load your exported groundTruth object and call the instanceSegmentationTrainingData function. The function converts polygon-labeled groundTruth objects into a training-ready datastore in a single call, automating polygon-to-mask conversion, bounding box computation, mask file writing, and datastore assembly.

load("gTruth.mat","gTruth")
ds = instanceSegmentationTrainingData(gTruth);

Verify the output format by previewing the datastore.

data = preview(ds)

Pass the datastore directly to trainSOLOV2 or trainMaskRCNN functions.

net = solov2("resnet50-coco",classNames,InputSize=inputSize);
detector = trainSOLOV2(ds,net,options);

You can also import COCO JSON annotations using groundTruthFromCOCO function and pass the resulting groundTruth object to the instanceSegmentationTrainingData function. If you want training data only for a select subset of polygon labels, use the selectLabelsByName function on the groundTruth object before calling instanceSegmentationTrainingData function.

Merge Data from Multiple Labeling Sessions

If you have ground truth data from multiple labeling sessions or video sources, pass an array of groundTruth objects. All objects must share the same polygon label names. If the groundTruth array does not have the same polygon label names, use the merge object function before calling the instanceSegmentationTrainingData function.

ds = instanceSegmentationTrainingData([gTruth1, gTruth2, gTruth3]);

Control Frame Subsampling and Output Location

When working with video-source groundTruth, use name-value arguments to control frame extraction.

ds = instanceSegmentationTrainingData(gTruth, ...
    SamplingFactor=5, ...
    WriteLocation="/data/training", ...
    ImageFormat="PNG", ...
    NamePrefix="scene1");

The SamplingFactor argument extracts every Nth frame to reduce redundant training images from video sequences. The WriteLocation argument specifies where the function writes mask MAT files and, for video sources, extracted image frames. For the full list of name-value arguments, see the instanceSegmentationTrainingData reference page.

Retrieve Per-Instance Attribute Data

If your groundTruth object includes attribute or sublabel annotations (for example, occluded or color), specify the optional second output argument while calling the instanceSegmentationTrainingData function.

[ds, arrds] = instanceSegmentationTrainingData(gTruth);

The second output argument arrds is an arrayDatastore of per-instance attribute structs, row-aligned with the masks and labels in ds. The arrds output is useful for filtering or analyzing training data by attribute.

Training Instance Segmentation Networks

Computer Vision Toolbox™ offers these functionalities for training an instance segmentation network, based on the type of network:

Instance Segmentation Network Type	Functionality
SOLOv2	solov2 – Create pretrained network trainSOLOV2 – Train network segmentObjects – Perform inference
Mask R-CNN	maskrcnn – Create pretrained network trainMaskRCNN – Train network segmentObjects – Perform inference

For examples on using the datastore to train instance segmentation networks, see:

• Perform Instance Segmentation Using SOLOv2

• Perform Instance Segmentation Using Mask R-CNN

For more information about datastores, see Datastores for Deep Learning (Deep Learning Toolbox).