Writing and executing tests

Scenarios must be invoked in order for Foretify to execute their behavior. For this purpose, Foretify provides an entry-point scenario, top.main(). You can extend top.main() to invoke either a user-defined or library scenario. This use of extension exemplifies the aspect-oriented nature of the OSC2 language, meaning you can modify the aspects of objects to suit a particular verification test.

The extension of top.main() to invoke a user-defined or library scenario is usually implemented in a separate file called a test. The separation of a test from a scenario reflects a separation of concerns—one of the most powerful features of OSC2. This distinction increases even more the reusability of OSC2 scenarios and tests. You can develop vendor-agnostic and execution platform-agnostic scenario libraries that can be reused later on by other companies and/or departments. It also allows regulatory agencies to implement tests or scenarios and provide those to vehicle producers.

Actors and scenarios have abstract properties or test parameters, whose values may change randomly from run to run. However, in order to meet a specific objective, you might need to control the values of these parameters. You control test parameters in the test file. Test files perform these additional functions:

Importing all the required OSC2 files.
Configuring the test by selecting a map, for example.
Invoking the user-defined or library scenario.

Note

This topic shows how to import OSC2 files for a test, then configure, constrain and execute the test. The complete code examples are included at the end of this topic. See Complete cut in and slow example 1 and Complete cut in and slow example 2.

Importing the OSC2 files

At least the first two lines of a typical test file are import statements. In this example, the test file begins by importing a simulator configuration file and the top-level file of a library scenario, sut.vehicle_cut_in_and_slow():

OSC2 code: import sim config and top OSC2 file

# ts_cut_in_and_slow.osc
#
import "$FTX_BASIC/exe_platforms/sumo_ssp/config/sumo_config.osc"
import "$FTX_PACKAGES/base_scenarios/scenarios/vehicle_cut_in/vehicle_cut_in_and_slow/vehicle_cut_in_and_slow_top.osc"

These files may in turn import other files. For example, a top-level scenario file often imports the main scenario implementation, any lower-level scenarios called by the main scenario, as well as checkers and metrics definitions. This file-by-file approach makes it easy to reuse scenarios and configure tests in multiple ways.

For more information on importing files, see import.

Configuring the test

You can configure some aspects of the test by extending the builtin configuration structs:

The ego_config struct lets you specify attributes related to the SUT (the Ego). For example, you can specify whether the SUT Support Package (DSP) should be launched. This support package translates messages between Foretify and the SUT. By default it is launched.
The sim_config struct lets you specify certain attributes of the simulater. For example, you can specify whether the Simulator Support Package (SSP) is launched, whether the simulator GUI opens, the simulation speed and so on.
The test_config struct lets you specify attributes of the test. For example, you can specify the map to use for the test, the maximum time allowed for a single run and so on.
The map_config struct lets you specify the attributes of the map. For example, you can specify the minimum speed limit required to consider a road as a highway or the default speed limit associated with roads that have no explicit speed limit.

Because there is no default map for tests, each test file must extend test_config to specify the map.

In this example, sim_config is also extended to specify that no additional time is added after scenario execution completes.

OSC2 code: configure the test

# ts_cut_in_and_slow.osc
#
extend test_config:
    set map = "$FTX_PACKAGES/maps/M77_FTX_highway_straight_long_road.xodr"
    set test_drain_time = 0second

For more detailed information on the OSC2 configuration options, see OSC2 configuration options.

You can also use a YAML file to configure the test flow, including the files to load, the map to use and so on.

For more detailed information on YAML configuration, see YAML configuration options.

Invoking a scenario

Foretify's builtin, top-level actor is top. The top.main() scenario is invoked automatically at the start of a run. You must extend top.main to invoke the top-level user-defined or library scenario. For example:

OSC2 code: extend top.main

# ts_cut_in_and_slow.osc
#
extend top.main:
    do serial:
        vehicle_cut_in_and_slow: sut.vehicle_cut_in_and_slow()

In this example, the sut.vehicle_cut_in_and_slow() scenario is specified with a label, vehicle_cut_in_and_slow. It is a good idea to add define a meaningful label when invoking a scenario because it simplifies referencing the scenario in constraints and other constructs. If no label is defined, Foretify assigns a label.

Applying constraints

When you analyze metrics in Foretify Manager, you may uncover some coverage holes because some corner cases have never been executed. Filling these holes might require tightening the constraints to direct the random generation process toward the missing behavior.

Additionally, tightening constraints makes the scenario narrower and simpler. This is useful in obtaining simple tests that are easier to debug. Constraints can also be used to avoid known issues with the SUT while those issues are being addressed.

Controlling basic vehicle attributes

You can set the attributes of the SUT and other vehicles using constraints. Constraints of interest include the vehicle color, which presumably affects sensor perception, as well as vehicle type, which affects physical properties like stopping distance and acceleration.

The vehicle type is set by a field called category that you can set to one of the following values: sedan, truck, bus, van, semi_trailer, trailer, four_wheel_drive.

The vehicle color is defined by the color field that you can set to one of the following values: white, black, red, green, blue, yellow, brown, pink, gray.

The following code extends the sut.vehicle_cut_in_and_slow() scenario to require that the cut_in_vehicle is a gray bus. The explicit constraint syntax uses the keyword keep with a Boolean expression:

OSC2 code: apply constraints in a scenario extension

# ts_cut_in_and_slow.osc
#
extend sut.vehicle_cut_in_and_slow:
    keep(cut_in_vehicle.category == bus)
    keep(cut_in_vehicle.color == gray)

You can also set constraints when invoking the scenario, instead of extending it. The following code sets the properties of cut_in_vehicle within the scenario invocation:

OSC2 code: apply constraints when invoking the top-level scenario

# ts2_vehicle_cut_in_and_slow.osc
#
extend top.main:
    do serial:
        vehicle_cut_in_and_slow: sut.vehicle_cut_in_and_slow()  with:
            keep(it.cut_in_vehicle.category == bus)
            keep(it.cut_in_vehicle.color == gray)

For more information on predefined vehicle attributes, see Vehicle actor.

Controlling cut-in-side

Let’s say you want to have more runs where the cut-in happens from the lane on the right of sut.car. You can easily achieve this by imposing a constraint over the scenario’s cut_in_side field, as in the following snippet:

OSC2 code: apply constraints when invoking the top-level scenario

# ts2_vehicle_cut_in_and_slow.osc
#
extend top.main:

    do serial:
        vehicle_cut_in_and_slow: sut.vehicle_cut_in_and_slow()  with:
            keep(it.cut_in_vehicle.category == bus)
            keep(it.cut_in_vehicle.color == gray)
            keep(it.cut_in_side == right)   # Force cut-in from the right

Within the keep() constraint, it.cut_in_side refers to the cut_in_side field of the sut.vehicle_cut_in_and_slow() scenario.

For more information on setting constraints, see Constraints.

Controlling relative speed

One way to make the maneuver more challenging is to make cut_in_vehicle decelerate more dramatically during the slow phase by adding the change_speed() modifier to the cut_in_vehicle.drive() scenario.

The scenario modifier in lets you add modifiers from outside of the scenario itself, instead of modifying the scenario description. In this example, the change_speed() modifier requires a relatively rapid slow down.

OSC2 code: apply constraints when invoking the top-level scenario

# ts_vehicle_cut_in_and_slow.osc
#
extend top.main:

    in vehicle_cut_in_and_slow.label(slow.cut_in_vehicle) with:
        change_speed([-50..-30]kph, run_mode: best_effort)

    do serial:
        vehicle_cut_in_and_slow: sut.vehicle_cut_in_and_slow()

The in modifier requires a scenario path (concatenated labels for scenario or modifier invocation) in order to identify which scenario gets the new members. Scenario and modifier invocation that have no user-defined labels get automatically generated labels with label().

In the above example, the scenario path concatenates the user-defined label vehicle_cut_in_and_slow with an automatically generated label, label(slow.cut_in_vehicle), thus referring to cut_in_vehicle.drive() in the slow phase of sut.vehicle_cut_in_and_slow.

The run_mode for change_speed() is set to best_effort. This means that when planning scenario execution, Foretify must choose a value in the specified range, [-50..-30]kph. However, during runtime, even if the change in speed of the cut_in_vehicle is outside of the specified range, the scenario will not fail to execute.

For more information see Automatic label computation.

Favoring higher-risk interaction

The vehicle that cuts in behaves according to a set of rules common to all vehicle agents. These rules limit their maneuvering and speed to model safe driving habits. Stressing the situation requires pushing the cutting in vehicle to maneuver more aggressively, thus challenging sut.car in unexpected ways. This can be achieved by limiting the amount of time allotted for the vehicle_cut_in phase defined in the vehicle_cut_in() scenario. (The vehicle_cut_in() scenario is a lower-level scenario invoked by sut.vehicle_cut_in_and_slow().)

In the following example, the in modifier is used to constrain the duration of the phase. The scenario path uses the user-defined labels vehicle_cut_in_and_slow, vehicle_cut_in and vehicle_cut_in. The constraint specifies the phase duration to be [2..5]s, forcing an aggressive cut-in.

OSC2 code: apply constraints when invoking the top-level scenario

# ts_vehicle_cut_in_and_slow.osc
#
extend top.main:
    in vehicle_cut_in_and_slow.label(slow.cut_in_vehicle) with:
        change_speed([-50..-30]kph, run_mode: best_effort)

    in vehicle_cut_in_and_slow.vehicle_cut_in.vehicle_cut_in with:
        duration([2..5]s, tolerance: 4s)

    do serial:
        vehicle_cut_in_and_slow: sut.vehicle_cut_in_and_slow()

The tolerance parameter is another way to require Foretify to plan a scenario execution whose duration falls within the specified range while allowing scenario execution to complete even if the duration is outside the range.

For more information, see Use best_effort or tolerance.

Scenario failures

Running the above test, some random seeds may cause a scenario failure — a system error indicating that the test did not reach the scenario goal. In these cases, the time allotted to the cut-in maneuver was not enough to carry it out. See Understanding and resolving issues for more information on how to resolve scenario failures.

Switching between maps

You may need to use a variety of maps in order to match a particular ODD (Operational Driving Domain). However, when you pick apparently suitable paths for the scenario from the map, the available paths may restrict the run in some ways. For example, an urban map with short segments and many intersections limits the maximal speed. Switching to an open road map removes this restriction, but adds others.

You can exercise scenarios with several different maps, using either a YAML or OSC2 configuration option. This example uses the OSC2 option.

OSC2 code: extend test_config

# ts2_vehicle_cut_in_and_slow.osc
#
extend test_config:
    set map = "$FTX_PACKAGES/maps/M73_FTX_highway.xodr"
    set test_drain_time = 0second

Map cropping

Map cropping is a feature that reduces the size of the map by limiting the graphical data to only include relevant areas based on the positions of objects in each frame.

By default, the map cropping feature is enabled. Use the following OSC code to disable map cropping:

extend map_config:
    set map_crop_enabled = false

Disabling the feature will save a full version of the map for each run.

Cropping alignment

Use the map_crop_alignment parameter to automatically refine the region for each run to include all relevant actors and margins. This parameter defines the grid that extends the cropping region along each axis.

For example, if the value is set to 100m, the cropping region on each axis will be aligned to the nearest 100m grid, so that the region ((430, 570), (940,1060)) is rounded to ((400, 500), (1000,1100)).

By default, the value is set to 50m, so the cropping region is aligned to a 50m grid.

For example:

extend map_config:
    set map_crop_alignment = 100m

This will align the calculated region to 100 meters on each axis.

Executing a test

You can execute a test using either Foretify Developer (Foretify's Graphical User Interface) or Foretify's Command-Line Interface (CLI).

If you like, you can run the Foretify Linter before or after executing a test to identify problematic use of the OSC2 language or enforce best practices for code used with Foretellix scenarios and libraries. See Using the Foretify Linter for more information.

Executing the test with the Command-Line Interface (CLI)

To execute the test using the CLI:

Create the ts_vehicle_cut_in_and_slow.osc file by copying the code shown below in Complete example 1 or Complete example 2.
Execute the following command in a command shell.
Shell command: invoke Foretify CLI
```
$ foretify --load ts_cut_in_and_slow.osc --run
```

To execute consecutive runs of the test using the CLI:

You can make consecutive runs of a test by either:

Specifying the initial seed and the number of runs to make, for example:

Shell command: make consecutive runs with incremented seed

# Make 10 consecutive runs and increment the seed each time starting with 43
$ foretify --load ts_cut_in_and_slow.osc --crun 10 --seed 43

Shell command: make consecutive runs with random seeds

# Make 10 consecutive runs and randomly generate a new seed each time
$ foretify --load ts_cut_in_and_slow.osc --crun 10 --seed random

Specifying a list of seeds.

Shell command: make consecutive runs with specific seeds

# Make 3 consecutive runs with the specified seeds
$ foretify --load ts_cut_in_and_slow.osc --crun_seeds "43,87,112"

A generation seed is a unique numeric value that, when specified for a run, causes Foretify to generate a specific set of values for the test. Foretify is repeatable, in other words, it reacts in exactly the same way to identical inputs. If the test platform is also repeatable, then the same run definition, including the test, seed and test configuration, will produce exactly the same cycle-by-cycle results.

For more information:

Using the Command-Line Interface

Executing the test with Foretify Developer

To execute the test using Foretify Developer:

Create the ts_vehicle_cut_in_and_slow.osc file by copying the code shown below in Complete example 1 or Complete example 2.
Set the Chrome browser as your default browser.

Note

Currently, Foretify requires the use of the Chrome browser.
Execute the following command in a command shell.
Shell command: invoke Foretify Developer
```
$ foretify --load ts_cut_in_and_slow.osc --gui
```
Foretify Developer opens as a Tab in your default Browser.
Click Run Test in the top-right corner of Foretify Developer.

For more information:

Loading, preparing and running tests

Loading the test into the Foretify Linter

To run the Foretify Linter:

Create the ts_vehicle_cut_in_and_slow.osc file by copying the code shown below in Complete example 1 or Complete example 2.
Execute the following command in a command shell.
Shell command: invoke Foretify Linter
```
$ foretify --load ts_cut_in_and_slow.osc --lint warn
```

For more information:

Using the Foretify Linter

Complete cut in and slow example 1

OSC2 code: ts1_cut_in_and_slow.osc
# ts1_vehicle_cut_in_and_slow.osc
#
import "$FTX_BASIC/exe_platforms/sumo_ssp/config/sumo_config.osc"
import "$FTX_PACKAGES/base_scenarios/scenarios/vehicle_cut_in/vehicle_cut_in_and_slow/vehicle_cut_in_and_slow_top.osc"

extend test_config:
    set map = "$FTX_PACKAGES/maps/straight_long_road.xodr"
    set test_drain_time = 0second

extend sut.vehicle_cut_in_and_slow:
    keep(cut_in_vehicle.vehicle_category == bus)
    keep(cut_in_vehicle.color == gray)
    keep(cut_in_side == right)

extend top.main:
    in vehicle_cut_in_and_slow.label(slow.cut_in_vehicle) with:
        change_speed([-50..-30]kph, run_mode: best_effort)
    in vehicle_cut_in_and_slow.vehicle_cut_in.vehicle_cut_in with:
        duration([2..7]s, tolerance: 4s)

    do serial:
        vehicle_cut_in_and_slow: sut.vehicle_cut_in_and_slow()

Lines 3 - 4 import the simulator configuration file and top-level library scenario file.
Lines 6 - 8 configure the test and select the map.
Lines 10 - 13 extend the scenario to constrain the cut_in_vehicle and cut_in_side.
Line 15 extends top.main().
Lines 16 - 17 constrain the speed with which the cut_in_vehicle changes lane.
Lines 18 - 19 constrain the duration of the cut-in phase.
Line 22 invokes the library scenario.

Complete cut in and slow example 2

OSC2 code: ts2_cut_in_and_slow.osc
# ts2_vehicle_cut_in_and_slow.osc
#
import "$FTX_BASIC/exe_platforms/sumo_ssp/config/sumo_config.osc"
import "$FTX_PACKAGES/base_scenarios/scenarios/vehicle_cut_in/vehicle_cut_in_and_slow/vehicle_cut_in_and_slow_top.osc"

extend test_config:
    set map = "$FTX_PACKAGES/maps/highway.xodr"
    set test_drain_time = 0second

extend top.main:
    do serial:
        vehicle_cut_in_and_slow: sut.vehicle_cut_in_and_slow()  with:
            keep(it.cut_in_vehicle.vehicle_category == bus)
            keep(it.cut_in_vehicle.color == gray)
            keep(it.vehicle_cut_in_at_essence.gen_cut_in_side == right)   # Force cut-in from the right

Lines 3 - 4 import the simulator configuration file and top-level library scenario file.
Lines 6 - 8 configure the test and select the map,
Line 10 extends top.main().
Lines 12 - 15 invoke the library scenario and constrain the cut_in_vehicle and cut_in_side.

Creating test templates for tests

Once you have created a test and run it several times to be sure it gives you the results you want, you can create a template for the test for use with FRun. See Creating test templates for an overview of FRun.