Use best_effort or tolerance
When writing a scenario, you typically use several modifiers such as speed(), position() or lane() to control an actor’s movement. During test execution, for each modifier of an actor’s movement that is not overridden with best_effort, Foretify confirms that the modifier’s requirement is satisfied. If a modifier is not satisfied, either the scenario time is extended or an incomplete_scenario error is issued.
To avoid unnecessarily extended or incomplete scenario executions, you can use one of the following methods:
-
Specify the tolerance parameter for the modifier.
-
Specify the run_mode: best_effort parameter for the modifier.
-
Use override(run_mode: best_effort) to override the modifier from outside the scenario block containing the modifier or from outside the scenario itself.
To use these methods properly, you need to understand the differences between them.
modifier(..., tolerance: <tolerance>)-
tolerance is an expression of type speed, length, acceleration or time specifying the maximum allowed deviation from the planned range. Tolerance is necessary because physical values received from the simulator might have errors in measurement. The default for the tolerance parameter depends on the modifier.
Generation always takes a value within the original range without considering tolerance. However, during test execution, when Foretify checks whether the modifier’s requirement is met, the tolerance parameter is considered, and the valid range is extended accordingly.
modifier(..., run_mode: best_effort)-
During test execution, Foretify does not check modifiers where the run_mode: best_effort parameter is specified. Even if the actor does not satisfy a modifier’s requirement, the scenario continues execution as if the modifier was satisfied.
override(..., run_mode: best_effort)-
You can override a modifier, and its tolerance parameter if specified, using override(). There is no runtime difference between the run_mode: best_effort parameter and override(). However, override() lets you change the modifier's run_mode from outside the scenario block where the modifier is specified.
Specify a best_effort parameter when the modifier is not critical for scenario intent.
You should specify a best_effort for any modifiers applied to the SUT. If you describe the expected behavior of the SUT by adding modifiers, Foretify can more easily plan scenario execution. However, since the SUT’s behavior is unpredictable, it is recommended to specify best_effort so that execution completes even if the SUT's actual behavior varies from the planned.
Also, specifying best_effort is recommended for modifiers of NPCs that are not critical for scenario intent. This facilitates mixing scenarios with potentially conflicting modifiers and reduces extended or failed scenario execution.
The following example shows a keep_lane() modifier on the SUT. If the SUT changes lanes, to avoid an obstacle, for example, the scenario fails.
import "$FTX_BASIC/exe_platforms/sumo_ssp/config/sumo_config.osc"
extend test_config:
set map = "$FTX_PACKAGES/maps/highway.xodr"
scenario sut.simple_drive:
do sut.car.drive() with:
keep_lane()
extend top.main:
do d1: sut.simple_drive()
This example shows how to set the expected behavior of the SUT, while allowing the scenario to complete execution even if the actual behavior is different.
import "$FTX_BASIC/exe_platforms/sumo_ssp/config/sumo_config.osc"
extend test_config:
set map = "$FTX_PACKAGES/maps/highway.xodr"
scenario sut.simple_drive:
do sut.car.drive() with:
keep_lane(run_mode: best_effort)
extend top.main:
do d1: sut.simple_drive()
Not yet implemented.
Specify a tolerance parameter for physical parameters critical to scenario intent
For modifiers that constrain physical parameters, specifying a tolerance or time_tolerance parameter requires Foretify to continue monitoring the modifier during runtime and only increase the valid range by the tolerance that you provide. For modifiers that are critical to implementing the scenario intent, specifying tolerance is important.
This example has two modifiers with physical parameters, lane() and position(), with no specified tolerance. If simple_vehicle has to travel in the same lane as sut.car in order to meet the position requirements, scenario execution may be unnecessarily extended or fail to complete.
import "$FTX_BASIC/exe_platforms/sumo_ssp/config/sumo_config.osc"
extend test_config:
set map = "$FTX_PACKAGES/maps/highway.xodr"
scenario vehicle.simple_drive:
do drive() with:
lane(side_of: sut.car)
position(time: 3second, ahead_of: sut.car, at: end)
extend top.main:
simple_vehicle: vehicle
do parallel(overlap: equal):
d1: simple_vehicle.simple_drive()
sut.car.drive()
This example defines the modifier that is critical to scenario intent, position(), with a tolerance parameter. The lane() modifier, which is less critical, is defined with run_mode: best_effort.
import "$FTX_BASIC/exe_platforms/sumo_ssp/config/sumo_config.osc"
extend test_config:
set map = "$FTX_PACKAGES/maps/highway.xodr"
scenario vehicle.simple_drive:
do drive() with:
lane(side_of: sut.car, run_mode: best_effort)
position(time: 3second, time_tolerance: 1s,
ahead_of: sut.car, at: end)
extend top.main:
simple_vehicle: vehicle
do parallel(overlap: equal):
d1: simple_vehicle.simple_drive()
sut.car.drive()
Not yet implemented.
Use override() when combining or mixing scenarios
Because run_mode: best_effort causes the tolerance parameter to be ignored, specifying both in the same scenario block doesn’t make any sense.
However, if you want to override the tolerance parameter from outside of the scenario block where the modifier is defined, you can use override() to do so. This is especially useful when you are calling a lower-level scenario from a higher one or executing two scenarios in parallel (mixing scenarios).
It does not make sense to define a tolerance parameter for a modifier and then override it in the same block. The tolerance parameter will be ignored.
import "$FTX_BASIC/exe_platforms/sumo_ssp/config/sumo_config.osc"
extend test_config:
set map = "$FTX_PACKAGES/maps/highway.xodr"
extend top.main:
simple_vehicle: vehicle
do d1: simple_vehicle.drive() with:
drive_speed: speed([30..50]kph, tolerance: 2kph)
override(drive_speed, run_mode: best_effort)
In this example, the speed() and distance() modifiers for the SUT are overridden because the SUT’s path might be blocked by the pedestrians. If tolerance parameters are specified for these modifiers, they will be ignored.
extend sut.sut_drive_with_persons:
persons_position: top.plain_object_group_position_along_drive(
persons_position_params, persons, sut_drive)
do drive_while_persons: parallel(start_to_start:0second, overlap:any):
sut_drive: sut.car.drive(duration: [3..10]second) with:
sut_drive_speed: speed(sut_speed)
# Since SUT's lane can be blocked by the persons,
# speed requirement might not be met
override(sut_drive_speed, run_mode: best_effort)
sut_distance: distance([70..100]m)
# Since SUT's lane can be blocked by the persons
# distance requirement might not be met
override(sut_distance,run_mode: best_effort)
persons_exists: persons.group_exists(persons_position_params)