Apply Labels to Unreal Scene Elements for Semantic Segmentation and Object Detection

The Simulation 3D Camera block provides an option to output semantic segmentation data from a scene. If you add new scene elements, or assets (such as traffic signs or roads), to a custom scene, then in the Unreal^® Editor, you must apply the correct ID to that element. This ID is known as a stencil ID. Without the correct stencil ID applied, the Simulation 3D Camera block does not recognize the scene element and does not display semantic segmentation data for it.

For example, this To Video Display window shows a stop sign that was added to a custom scene. The Semantic Segmentation Display window does not display the stop sign, because the stop sign is missing a stencil ID.

You can also use these stencil ID labels to assign IDs to objects detected by probabilistic sensors, such as those modeled by the Simulation 3D Probabilistic Radar and Simulation 3D Vision Detection Generator blocks.

To apply a stencil ID label to a scene element, follow these steps:

1. Open the Unreal Editor from a Simulink^® model that is configured to simulate in the 3D environment. For more details, see Customize Scenes Using Simulink and Unreal Editor.

2. In the editor window, select the scene element with the missing stencil ID.

3. On the Details pane on the right, in the Rendering section, select Render CustomDepth Pass.

   If you do not see this option, click the Show Advanced expander to show all rendering options.

4. In the CustomDepth Stencil Value box, enter the stencil ID that corresponds to the asset. If you are adding an asset to a scene from the Automated Driving Toolbox™ Interface for Unreal Engine^® Projects support package, then enter the stencil ID corresponding to that asset type, as shown in the table. If you are adding assets other than the ones shown, then you can assign them to unused IDs. If you do not assign a stencil ID to an asset, then the Unreal Editor assigns that asset an ID of 0.

   Note

   The Simulation 3D Camera block does not support the output of semantic segmentation data for lane markings. Even if you assign a stencil ID to lane markings, the block ignores this setting.

   ID	Type
   0	None/default
   1	Building
   2	Not used
   3	Other
   4	Pedestrians
   5	Pole
   6	Lane Markings
   7	Road
   8	Sidewalk
   9	Vegetation
   10	Vehicle
   11	Not used
   12	Generic traffic sign
   13	Stop sign
   14	Yield sign
   15	Speed limit sign
   16	Weight limit sign
   17-18	Not used
   19	Left and right arrow warning sign
   20	Left chevron warning sign
   21	Right chevron warning sign
   22	Not used
   23	Right one-way sign
   24	Not used
   25	School bus only sign
   26-38	Not used
   39	Crosswalk sign
   40	Not used
   41	Traffic signal
   42	Curve right warning sign
   43	Curve left warning sign
   44	Up right arrow warning sign
   45-47	Not used
   48	Railroad crossing sign
   49	Street sign
   50	Roundabout warning sign
   51	Fire hydrant
   52	Exit sign
   53	Bike lane sign
   54-56	Not used
   57	Sky
   58	Curb
   59	Flyover ramp
   60	Road guard rail
   61	Bicyclist
   62-66	Not used
   67	Deer
   68-70	Not used
   71	Barricade
   72	Motorcycle
   73-255	Not used

   For example, for a stop sign that is missing a stencil ID, enter 13.

   Tip

   If you are adding stencil ID for scene elements of the same type, you can copy (Ctrl+C) and paste (Ctrl+V) the element with the added stencil ID. The copied scene element includes the stencil ID.

5. Visually verify that the correct stencil ID shows by using the custom stencil view. In the top-left corner of the editor window, click and select Buffer Visualization > Custom Stencil. The scene displays the stencil IDs specified for each scene element. For example, if you added the correct stencil ID to a stop sign (13) then the editor window, the stop sign displays a stencil ID value of 13.

   

   • If you did not set a stencil ID value for a scene element, then the element appears in black and displays no stencil ID.

   • If you did not select CustomDepth Stencil Value, then the scene element does not appear at all in this view.

6. Turn off the custom stencil ID view. In the top-left corner of the editor window, click Buffer Visualization and then select Lit.

To display semantic segmentation data for your custom labels, follow these steps:

1. Set up your Simulink model to display semantic segmentation data from a Simulation 3D Camera block. For an example setup, see Depth and Semantic Segmentation Visualization Using Unreal Engine Simulation.

2. Run the simulation and verify that the Simulation 3D Camera block outputs the correct data. For example, here is the Semantic Segmentation Display window with the correct stencil ID applied to a stop sign.