Automate Testing for Scenario Variants of AEB System
This example shows how to assess the functionality of an autonomous emergency braking (AEB) application by varying the European New Car Assessment Programme (Euro NCAP) Car-to-Pedestrian Nearside Child (CPNC) driving scenario. This example builds on the Autonomous Emergency Braking with Sensor Fusion example.
Autonomous emergency braking is an advanced active safety system that helps drivers avoid or mitigate collisions with other vehicles. The test space for autonomous driving functions is incredibly large, and it is not feasible to validate large numbers of scenarios with real-world driving tests. This example enables you to perform scenario-based testing of an AEB system by generating multiple scenario variants from a seed scenario.
The example shows how to vary ego speed and collision point parameters to generate multiple variants of the Euro NCAP CPNC test scenario. The example also shows how to use iterative testing methodology in Simulink® Test Manager to test the generated variants. In this example, you:
Explore the test bench model — The model contains modules for the sensors and environment, sensor fusion and tracking, decision logic, controls, and vehicle dynamics.
Review seed scenario and generate variants — Review the CPNC seed scenario and generate variants of the seed scenario by varying the ego speed and the collision point between the ego vehicle and pedestrian.
Perform iterative testing with scenario variants — Configure Test Manager to simulate the AEB test bench model for each scenario variant using scripted iterative testing with Simulink Test™, assess success criteria, and report results.
Visualize the results and review the generated report — Plot a grid-based visualization to review and analyze the results from iterative testing. You can also export the simulation results to a PDF and review the generated report.
Explore Test Bench Model
This example reuses the
AEBTestBench model from the Autonomous Emergency Braking with Sensor Fusion example.
To explore the test bench model, open a working copy of the project example files. MATLAB® copies the files to an example folder so that you can edit them.
To reduce Command Window output, turn off model predictive controller (MPC) update messages.
Open the system-level simulation test bench model.
This test bench model has these modules:
Sensors and Environment— Subsystem that specifies the road, actors, camera, and radar sensor used for simulation.
Sensor Fusion and Tracking— Algorithm model that fuses vehicle detections from the camera to those from the radar sensor.
AEB Decision Logic— Algorithm model that specifies the lateral and longitudinal decision logic that provides most important object (MIO) related information and ego vehicle reference path information to the controller.
AEB Controller— Algorithm model that specifies the steering angle and acceleration controls.
Vehicle Dynamics— Subsystem that specifies the dynamic model of the ego vehicle.
Metrics Assessment— Subsystem that assesses system-level behavior.
For more details on these components and simulating the test bench model, see the Autonomous Emergency Braking with Sensor Fusion example.
In this example, your focus is on automating the simulation runs for multiple variants of a seed scenario. You create the seed scenario using the Driving Scenario Designer app.
Review Seed Scenario and Generate Variants
helperCreateSeedScenario function generates a cuboid scenario that is compatible with the
AEBTestBench model. This is an open-loop scenario containing two stationary target vehicles on a straight road and a child pedestrian. This is a Euro NCAP CPNC scenario, in which the ego vehicle collides with a pedestrian child who is running behind the obstruction from the nearside. The frontal structure of the vehicle hits the pedestrian.
Plot the open-loop scenario to see the interactions of the ego vehicle and the pedestrian child target.
hFigSeedScenario = helperPlotScenario("helperCreateSeedScenario");
The ego vehicle, which is not under closed-loop control, collides with the pedestrian child. The goal of the closed-loop system is to avoid or mitigate the collision with the pedestrian actor. In the
AEBTestBench model, the ego vehicle has the same initial speed and initial position as in the open-loop scenario.
Close the figure.
Generate Variants of Seed Scenario
helperCreateSeedScenario function creates a seed scenario in which the speed of the ego vehicle is 60 km/hr, and the collision point is at the front-left corner of the ego vehicle. To test this AEB CPNC scenario per Euro NCAP test protocols, you create multiple variants of the seed scenario by varying the ego speed and collision point. Use the
helperCreateSeedScenario function to create the seed scenario.
[seedScenario,egoVehicle] = helperCreateSeedScenario;
getScenarioDescriptor function to create a
scenarioDescriptor object that has the properties of the seed scenario. You can modify the properties of this object to generate scenario variations.
% Get the scenario descriptor from the seed scenario scenarioDescriptor = getScenarioDescriptor(seedScenario);
Define the IDs of the ego vehicle and the target actor that collides with the ego vehicle.
egoID = egoVehicle.ActorID; targetID = 2;
To generate scenario variations, the ego vehicle in the seed scenario must satisfy these criteria:
The ego vehicle must collide with the target vehicle.
The ego vehicle must have consistent motion properties.
The ego trajectory must have at least three waypoints.
The ego vehicle must travel at a constant speed.
helperCheckScenario function validates the seed scenario against these criteria. If any of the checks fail, the function returns a value of
0 indicating that you must modify the scenario before using it to generate variations variations.
helperPerformScenarioCollision function modifies the scenario based on the conditions specified in Generate Scenario Variants for Testing AEB Pedestrian Systems example.
% Validate the seed scenario information stored in scenarioDescriptor. checkStatus = helperCheckScenario(egoID,targetID,scenarioDescriptor); if ~checkStatus % if scenario check failed, modify the scenario according to requirements scenarioDescriptor = helperPerformScenarioCollision(egoID,targetID,scenarioDescriptor,method="WaitTime"); end
To generate variants of the seed scenario, use the
helperGenerateScenarioVariant function. The
helperGenerateScenarioVariant function uses these input arguments: the
scenarioDescriptor object, the ego vehicle ID, the target actor ID, a new speed for the ego vehicle, and a new collision point.
Specify the new ego speed in m/s. To convert 20 km/h to m/s, divide by 3.6. This results in a speed value of approximately 5.5 m/s.
egoNewSpeed = 20 * 1/3.6; % m/s
Define the new collision point, as specified in the Generate Scenario Variants for Testing AEB Pedestrian Systems example, which is required for scenario variant generation. For a collision point at the center of the front of the ego vehicle front, specify
newCollisionPoint = 0.5;
Generate the variant, using the
helperGenerateScenarioVariant function, with the ego vehicle traveling at 20 km/h and hitting the pedestrian at the center of the front of the vehicle.
scenarioVariant = helperGenerateScenarioVariant(scenarioDescriptor,egoID,targetID,egoNewSpeed,newCollisionPoint);
Review the generated variant scenario using the Driving Scenario Designer app.
Notice that the generated variant has an ego speed of 5.5 m/s, and the collision occurs at the center of the front of the ego vehicle. You can use this scenario with the
AEBTestBench model to perform closed-loop testing.
Simulink Test enables you to automate the generation and testing of scenario variants with the closed-loop model. Using Simulink Test, you can generate a large number of variations from a seed scenario and catch the failures.
Perform Iterative Testing with Scenario Variants
This example includes a Test Manager configuration for automating testing of the AEB application with the generated variants. Open the
AEBScenarioVariationTest.mldatx test file in the Test Manager.
sltestmgr testFile = sltest.testmanager.load("AEBScenarioVariationTest");
The Simulink scripted iteration framework enables you to automate scenario variant generation and closed-loop testing using the test bench model. The iterative testing script in the
AEBScenarioVariationTest.mldatx file describes how to combine scenario variant generation and closed-loop testing of the AEB application.
setVariable function sets these parameters for each iteration:
v_set— Set Velocity of the ego vehicle.
scenario— Driving Scenario object.
egoVehDyn— Initialization parameters for
Vehicle Dynamicssubsystem, specified as a structure.
addIteration function adds test iterations. For more information on creating scripted iterations, see the Test Iterations (Simulink Test).
After simulating the test case, the Test Manager uses the
helperPlotAEBResults function from the CLEANUP callback to generate the post-simulation plots. For more information on these plots, see Autonomous Emergency Braking with Sensor Fusion example.
Run and Explore Results for CPNC Scenario Variants
To test the system-level model with the generated variants from Simulink Test, use this code:
resultSet = run(testFile); testFileResults = getTestFileResults(resultSet); testSuiteResults = getTestSuiteResults(testFileResults); testCaseResults = getTestCaseResults(testSuiteResults); testIterationResults = getIterationResults(testCaseResults);
Visualize Results and Review Generated Report
testIterationResults output contains the results for all the iterations. To plot the summary of results in the form of a color grid, use the
helperPlotAEBVariantResults function. This helper function plots the grid results with specific colors for each cell that indicate the pass or fail status of closed-loop simulation of the
AEBTestBench model with the generated scenario variants.
In the result grid, the columns represent the collision point values of
0.7. The rows represent nine ego speed values ranging from 10 km/h to 50 km/h in steps of 5 km/h. For each value of ego speed, there are four possible collision points, which results in a total of 36 variations. The right plot in the figure shows the color lookup table, where green shows very significant speed reduction of the ego vehicle at collision time, and red represents low speed reduction at collision time. The speed reduction of the ego vehicle is the difference between the initial ego speed and the ego speed at the time of collision. If there is no collision, the speed reduction is the difference between the initial speed and the ego speed at the end of the simulation.
Each cell of the grid shows the final ego speed using a color that corresponds to the amount of speed reduction. Notice that all cells of the grid are green, which indicates that the algorithm in the AEB test bench passed all 36 variations of the CPNC scenario.
Use this code to generate the report, for further analysis of specific iterations:
sltest.testmanager.report(testIterationResults,"Report.pdf", ... Title="Automate Testing for CPNC Scenario Variants of AEB System", ... IncludeMATLABFigures=true, ... IncludeErrorMessages=true, ... IncludeTestResults=0, ... LaunchReport=true);
Review Generated Report
Report.pdf. Observe that the
Test environment section shows the platform on which the test is run and the MATLAB version used for testing. The
Summary section shows the outcome of the test and duration of the simulation, in seconds. The
Results section shows pass or fail results based on the assessment criteria. This section also shows the plots logged from the
Enable MPC update messages.
- Vehicle Body 3DOF (Vehicle Dynamics Blockset) | Driving Radar Data Generator | Vision Detection Generator