Apply Labels to Unreal Scene Elements for Semantic Segmentation and Object Detection

The Simulation 3D Camera block provides an option to output semantic segmentation data from an Unreal^® scene. If you add new scene elements, or assets, to a custom scene, then you must apply the correct ID to that element in the Unreal Editor. This ID is known as a stencil ID. Without the correct stencil ID applied, the Simulation 3D Camera block does not recognize the scene element and does not display semantic segmentation data for it.

For example, this Camera Display window shows a stop sign that was added to a custom scene. The Semantic Segmentation Display window does not display the stop sign, because the stop sign is missing a stencil ID.

You can apply a stencil ID to a new scene element using Unreal Editor or the sim3d.Actor object. You can use the ActorClassId argument of the sim3d.Actor object to apply the ID.

To apply a stencil ID label to a scene element in Unreal Editor:

1. Open the Unreal Editor from a Simulink^® model that is configured to simulate in the 3D environment. .

2. Add an asset to the scene. For more details, see Add Assets to Scene.

3. In the editor window, select the scene element with the missing stencil ID.

4. On the Details pane on the right, in the Rendering section, select Render CustomDepth Pass.

   If you do not see this option, expand Advanced and click the Show Advanced to show all rendering options.

5. In the CustomDepth Stencil Value box, enter the stencil ID that corresponds to the asset. If you are adding an asset to a scene from the Simulink 3D Animation™ Interface for Unreal Engine^® Projects support package, then enter the stencil ID that corresponds to that asset type, as shown in the table. If you are adding assets other than the ones shown, then you can assign them to unused IDs. If you do not assign a stencil ID to an asset, then the Unreal Editor assigns that asset an ID of 0.

   The table lists the stencil IDs assigned to the scene elements in the prebuilt scenes. For example, for a stop sign that is missing a stencil ID, enter 13.

   ID	Type
   0	None/default
   1	Building
   2	Not used
   3	Other
   4	Pedestrians
   5	Pole
   6	Lane markings
   7	Road
   8	Sidewalk
   9	Vegetation
   10	Vehicle
   11	Not used
   12	Generic traffic sign
   13	Stop sign
   14	Yield sign
   15	Speed limit sign
   16	Weight limit sign
   17-18	Not used
   19	Left and right arrow warning sign
   20	Left chevron warning sign
   21	Right chevron warning sign
   22	Not used
   23	Right one-way sign
   24	Not used
   25	School bus only sign
   26-38	Not used
   39	Crosswalk sign
   40	Not used
   41	Traffic signal
   42	Curve right warning sign
   43	Curve left warning sign
   44	Up right arrow warning sign
   45-47	Not used
   48	Railroad crossing sign
   49	Street sign
   50	Roundabout warning sign
   51	Fire hydrant
   52	Exit sign
   53	Bike lane sign
   54-56	Not used
   57	Sky
   58	Curb
   59	Flyover ramp
   60	Road guard rail
   61	Bicyclist
   62-66	Not used
   67	Deer
   68-70	Not used
   71	Barricade
   72	Motorcycle
   73-255	Not used

   Tip

   If you are adding stencil ID for scene elements of the same type, you can copy (Ctrl+C) and paste (Ctrl+V) the element with the added stencil ID. The copied scene element includes the stencil ID.

6. Visually verify that the correct stencil ID shows by using the custom stencil view. In the top-left corner of the editor window, click and select Buffer Visualization > Custom Stencil. The scene displays the stencil ID specified for each scene element. For example, if you added the correct stencil ID to a stop sign (13), then in the editor window, the stop sign displays a stencil ID value of 13.

   

   • If you do not set a stencil ID value for a scene element, then the element appears in black and displays no stencil ID.

   • If you do not select CustomDepth Stencil Value, then the scene element does not appear in this view.

7. Turn off the custom stencil ID view. In the top-left corner of the editor window, click Buffer Visualization, and then select Lit.

To display semantic segmentation data for your custom labels:

1. Set up your Simulink model to display semantic segmentation data from a Simulation 3D Camera block. .

2. Run the simulation and verify that the Simulation 3D Camera block outputs the correct data. For example, this Semantic Segmentation Display window shows the correct stencil ID applied to a stop sign.