Understanding and resolving issues
One of the main verification challenges is to determine — in the most efficient manner — whether the SUT performed correctly. For example, you want to know if the SUT:
- Did not meet the minimum legal highway speed.
- Failed to keep a proper distance from pedestrians.
- Drove off the road.
Checkers are OSC2 entities that define the ways that the SUT should or should not behave. The simplest checkers define a failure response if the SUT fails to meet a Boolean condition. For example, if the SUT should never go faster than 80 kph, you can write a checker like this:
on (sut.car.state.speed > 80kph):
call sut_error(sut_too_fast,
"SUT speed too high: $(sut.car.state.speed)")
See complete example.
Foretify provides some standard checkers, but you can control whether Foretify ignores these checkers or identifies a checker failure as an issue. You can also define custom checkers to capture issues specific to your SUT.
When a checker fails, or something else goes wrong during a run, Foretify’s issue handling mechanism records and categorizes information about the issue in top.issues:
- top.issues.all_issues is list of all issues that have occurred since the start of the run.
- top.issues.main_issue is the issue with the highest severity that has occurred up to this point in the run.
- top.issues.curr is the issue currently being handled.
Issues are defined with a severity level ranging from ignore to error_stop_now. After the current issue is handled, it replaces the main issue if its severity is greater than the previous one. Additionally, issues with severity level error_continue and above trigger a run failure.
Foretify Manager aggregates these metrics across a set of test suite results and facilitates analysis of common issues among the runs. This process of analyzing and resolving issues is called verdict analysis. Foretify Manager provides a Triage view for verdict analysis. See Triaging test suite results for details.
Understanding issue definition
An issue is captured by a struct of type issue_struct containing the following fields:
- severity
- category
- kind
Depending on the kind of issue, additional information may be collected:
- details
- normalized_details
- full_message
- time
- modification_reason
Severity
This field is of the enumerated type issue_severity and has one of the following values:
| Issue | Description |
|---|---|
| ignore | This issue will not be reported. |
| info | This issue is available for review. |
| warning | This issue should be reviewed. |
| error_continue | This issue is an error, but the run should continue until completion. error_continue issues are grouped into the Errors category in run results. |
| error | This issue is an error, but the run should continue until the end of the cycle. error issues are grouped into the Errors category in run results. |
| error_stop_now | This issue is an error, and the run should stop immediately. error_stop_now issues are grouped into the Errors category in run results. |
Notes
- issue_severity is used for defining and modifying the severity of checks in OSC2 and as part of the issue_struct defined above. The severity level reported during a run does not always match the enumerated types in the definition above. For example, the severity levels, error_continue, error, and error_stop_now, are grouped into the category, Errors, in run results.
- Because these severity levels are used by Foretify and Foretify Manager to handle issues in various ways, it is recommended not to extend the issue_severity type.
- The issue_severity defined in OSC is displayed in Foretify Manager under the Main Original Issue Severity attribute. You can add this attribute using the column chooser or filter it using the run filter. Foretify Manager also provides a more general attribute, Main Issue Severity, which maps all errors to a single Error value.
Category
This field is of the enumerated type issue_category and has one of the following values:
| Category | Description |
|---|---|
| sut | These issues occur when the SUT behaves in unsafe or unexpected ways. |
| scenario_completion | These issues occur when the intended scenario conditions cannot be met and the scenario exits before completion. |
| other | These issues include all other types of issues such as exceptions, assertions, static analysis contradictions and so on. |
Kind
This field is of the enumerated type issue_kind and has various predefined values as described in the following chapters.
Note
There are additional issue kinds used by scenario-level checks. These are documented in the specific scenario documentation.
Note
When you create a checker, it is recommended to extend the issue_kind type with a unique name for the checker. This way you can easily identify runs with the issue you have defined. See Defining checkers.
Issue kinds for load, compilation, and generation-related errors
load_failed: Test file load failed due to reasons such as file not found. See load_failed for instructions on how to solve these issues.
compilation_error: Compilation failed due to an OSC2 coding error. See compilation_error for instructions on how to solve these issues.
contradiction: A contradiction error occurs when Foretify cannot assign a value to the attribute of an object because of conflicting constraints. To help you debug constraint contradictions, Foretify prints messages about the contradiction. Issues of error-kind contradiction are thrown when a contradiction is detected during the modeling stage, if a contradiction is detected at the solving stage a solver_failure will be thrown. see contradiction detection for more details on contradictions and how to resolve them.
solver_failure: Generation could not find a solution during the solving stage after the last retry. This could be due to contradictions, checks in the post-generation phase, or a combination of these. See documentation for more details on this issue, reasons, and resolutions. See Understanding generation for a full description of the different generation stages, the issues happening in each of those, and how to deal with them.
Issue kinds for scenario completion errors
incomplete_scenario: The scenario didn't execute as expected since the runtime monitor identified that a certain constraint could not be held. This can happen due to deviation from the plan (an actor behaves differently than expected) or coding issues in the scenario. See incomplete_scenario for a more detailed explanation.
unavoidable_incomplete_scenario: The scenario didn't complete as expected for reasons unrelated to Foretify (i.e. due to topology, the actors end up in a driving situation, in which they cannot complete their required drives). Used for well-defined cases in which the incomplete scenario is expected and explained. For more information on incomplete scenarios, see incomplete_scenario.
unfeasible_plan: Foretify succeeds in creating a plan, but this plan is evaluated as not feasible on its initial state in the runtime evaluator, as fully evaluating the plan can be achieved during run-time only, while the scenario planner makes some abstractions. See unfeasible_plan for a more detailed explanation.
Issue kinds for collisions
collision_in_hybrid_control: A collision occured while the SUT was engaged driving the vehicle in parallel to the Foretellix Human Driver Model (usually ADAS type SUT).
collision_in_sut_control: A collision occured while the SUT was in full control of the vehicle.
collision_in_manual_control: A collision occured while the SUT was not in control of the vehicle (the vehicle was fully controlled by the Foretellix Human Driver Model).
See details in collision-error-messages for more details on these issue kinds and how to handle these.
Issue kinds for timeouts and internal failures
exception: An exception has occurred in Foretify execution code. This is an internal error that needs further investigation; hence, please report a bug ticket to Foretellix.
assertion: An internal assertion was triggered using assertion_error_if or assertion_error. Unless this is an assertion intentionally thrown in user code (see assertion_error ) this is an internal error that needs further investigation; hence, please report a bug ticket to Foretellix.
run_timeout: Foretify terminates a run if there is some delay in generation or execution extending the whole run duration beyond the specified run_timeout parameter. If you see many incomplete runs, you can set a timeout to allow more time for the run to continue. See here to know how to invoke Foretify with a different run_timeout and how to handle run timeouts.
internal_error: An internal error can be triggered by two different root causes: Internal_error is used for unexpected errors (in this case, please report a bug ticket to Foretellix) or keep_alive timeout issues, this can be differentiated by looking at the error message. Foretify issues a keep_alive timeout, when, by default, Foretify waits for up to 60 seconds for a keep-alive message from its runtime engine. This engine operates during scenario execution, not during generation. If the runtime engine does not respond with a message within the defined timeout, Foretify terminates the run. See here for more details.
driver_behavior_invalid_objective: An issue with one of the driver behavior models occurred, which produces an invalid objective. This is an internal error that needs further investigation; hence, please report a bug ticket to Foretellix.
off_planned_path: The vehicle is too far from the path planned by Foretify. For more details about configuring this check, see vehicle-policy-max_road_offset-checker.
- If this issue occurs for a vehicle controlled by Foretify, report it as a bug ticket to Foretellix.
- If this issue occurs for an SUT, investigate the reason why the SUT diverged from the path planned by Foretify.
Issue kinds for other failures
simulation: A common simulation issue can arise if you forget to configure the simulator or other execution platform. You need to configure a simulator in your test file to fix this issue. For example, this can also happen if a specific feature is used in Foretify, such as traffic lights, which is not supported by the simulator. See simulation for more details.
default_collect_kind: A default issue kind for implementations of collect without a defined issue_kind. See collect for more details.
invalid_traffic_light_id: In the OSC2 scenario code, either the traffic_light.set_light_state method or map.set_tl_direction_agnostic_state() was called with an invalid traffic light ID that is not within the valid range. Review the traffic_light_id argument passed to the relevant action or method in the scenario. It should be noted that internal_road.traffic_light_id may be ‘-1’ if internal_road is not controlled by a traffic light.
abort: The user aborted the run using the stop button.
driver_invalid_shape_trajectory: The provided trajectory has discontinuity issues (abrupt change in position or speed which is physically impossible for a vehicle to perform). Critical for exact trajectory execution, but not critical in the case of behavioral execution, as the motion is smoothened. Check the defined shape.
dynamic_calibration_error: If the Driver fails to execute the required trajectory, this might indicate an issue in the controller calibration for that vehicle model. Please check the calibration of the vehicle models and contact your AE for the support if needed.
out_of_odd: ODD monitoring checks the dynamic state of the vehicle (its local acceleration, orientation angles, angular rates, and slip angles). Details on each check and error message as well as parameters to tune these checkers are documented in odd-checkers. We recommend being very careful when changing these, as it can cause unwanted behavior in the Foretellix Driver.
unsupported_OSC2.0_constructs: Nested lists with non-deterministic sizes are not supported. See Nested lists with non-deterministic sizes for more details.
Issue kinds for global checkers
See Global checkers for vehicles for more details on the following issue kinds issue_kind.
vehicle Physical Checks
speed_physical: The speed_physical checks that the vehicle's speed does not exceed or fall below the physical capability of the vehicle. See vehicle-physical-speed for more details.
acceleration_physical: The acceleration_physical checks that the vehicle's acceleration does not exceed or fall below the physical capability of the vehicle. See vehicle-physical-acceleration for more details.
turn_physical: The turn_physical determines whether the vehicle's angular and lateral movements are feasible. See vehicle-physical-turning-radius for more details.
Vehicle Policy Checks
speed_limit: The speed_limit checks that the vehicle does not exceed the legal speed limit indicated on the map. See vehicle-policy-legal-speed-limit for more details.
speed_policy: The speed_policy checks that the vehicle did not exceed the declared value vehicle.policy.max_speed or fall below the vehicle.policy.min_speed (for driving in reverse). See vehicle-policy-speed for more details.
acceleration_policy: The acceleration_policy checks that the vehicle did not exceed the declared value of vehicle.policy.max_acceleration or fall below the vehicle.policy.min_acceleration (for braking). See vehicle-policy-acceleration for more details.
perceived_teleport: If a vehicle's momentary speed exceeds the speed set in the vehicle.policy.max_speed by a factor of two or more, a perceived_teleport error is raised. See vehicle-policy-no-teleport for more details.
jerk_policy: The jerk_policy checks that the vehicle does not experience a sudden and dramatic decrease or increase in acceleration. See vehicle-policy-jerk for more details.
jitter_policy: The jitter_policy checks that there are fewer than four noticeable steers at a rate of more than one steer per two seconds. A noticeable steer is defined as 0.7m. For more details, see vehicle-policy-jitter.
minimum_highway_vru_distance_policy: The minimum_highway_speed_policy checks that when the SUT vehicle is on a highway and controlled by an AV or an ADAS function, it does not drive slower than vehicle.policy.minimum_highway_speed unless there is a reason, such as a car or a junction ahead. See vehicle-policy-minimum-highway-speed for more details.
unplanned_standing_time_policy: The unplanned_standing_time_policy checks that a vehicle does not stand still while moving for more than vehicle.policy.unplanned_standing_time unless there is a reason, such as a car or a junction ahead. See vehicle-policy-unplanned-standing-time for more details.
car_does_not_move: The car_does_not_move checks that the SUT vehicle moves at least the distance specified by policy.min_total_distance_to_indicate_movement within the time specified by policy.max_does_not_move_time. See vehicle-policy-does-not-move for more details.
ttc_policy: The ttc_policy is triggered if the time to collision is less than policy.too_close_TTC. See vehicle-policy-warn-if-too-close-to-collision for more details.
off_road: An off_road issue with severity error is thrown at the end of each interval during which the vehicle crossed the road boundary by more than the distance specified by error_threshold / junction_error_threshold of vehicle.global_checkers.off_road_checker. An off_road issue with severity warning is thrown at the end of each interval during which the bounding box of the vehicle crosses the road boundary by more than the distance specified by warning_threshold / junction_warning_threshold of vehicle.global_checkers.off_road_checker. See [Vehicle policy: off_road][vehicle-policy-off_road] for more details.
off_lane: The off_lane checks that the vehicle does not drive in a lane not intended for its use, such as a cycling lane. Deviations of up to 25% of the vehicle's width are allowed when a vehicle is dynamically controlled by Foretify or such control has ended without enough time for the AV or ADAS function to react. See vehicle-policy-off_lane for more details.
stopping_for_stop_sign: This error is raised when a vehicle does not stop in front of a stop sign for a parameterized amount of time. See Junction checkers for more details.
Identifying the main issue
At the end of the run, Foretify prints the main issue that occurred during the run, for example:
[INFO] [MAIN] [11.980] Main issue:
Other warning (acceleration_physical)
Foretify determines the main issue by comparing the severity of each issue as it occurs to the severity of the currently identified main issue. If the severity of the new issue is greater, it is then identified as the main issue.
Understanding test phases
In order to understand an issue, it helps to know when the issue is identified. Foretify identifies various kinds of issues during each of the four test phases, as described below.
Compilation
Foretify tries to load all the files required for the test. This phase identifies issues such as:
- Missing OSC2 files
- Parsing errors
- An invalid path for map
- An invalid map file
- Missing external method implementations
- Missing libraries
In addition, if you have invoked Foretify with the --lint option, some linter rules are checked in this phase. See Using the Foretify Linter and Foretify Linter rules.
Static analysis
Foretify tries to build a planning problem by combining all the scenarios and modifiers defined in the test with:
- Implicit constraints, such as: distance = speed * duration.
- Internal Foretify constraints, such as: the duration of a scenario has to be a multiple of the simulator step time.
This phase identifies contradictions between the combined constraints and fails if a required road element is not found on the map. For example, if a scenario requires driving on a highway_entry element but the loaded map has no instance of highway_entry, this phase fails.
Additional linter rules are checked in this phase if you have invoked Foretify with the --lint option.
Generation
Foretify chooses a possible solution to the planning problem. The solution chosen may vary according to the seed. The solution includes planned positions and speeds for all actors in the scenario, including the SUT.
Based on the chosen solution, for each actor controlled by Foretify, a list of objectives is created. Each objective is defined as a required position and speed at a specific time. Some of the objectives are absolute and some are relative. For example, if an NPC is required to drive faster than the SUT, a relative objective is created.
This phase is also called Planning.
Execution
Foretify tries to execute the test as planned and achieve the list of objectives for every actor controlled by Foretify. In addition, during the execution phase, for each modifier of drive that is not overridden, Foretify confirms that it is satisfied. If an objective or a modifier is not satisfied, either the scenario time is extended or an incomplete_scenario error is issued.
Other errors that occur during execution include collisions, OSC2 code execution errors, and issues identified by global checkers, ODD checkers, and scenario-specific checkers.
Creating issues
Foretify includes code that generates issues. For example, built-in checkers, incomplete scenarios and other errors generate issues. In addition, you can write your own checks that raise issues using the methods described in this section. You can also create watcher checkers which provide a robust infrastructure for writing custom checks and for modifying issues.
This section describes the sut_issue struct and the methods for creating various types of issues. These methods are defined under the top struct. If you are calling these methods from C++ code, you should append top() before the method call.
issue struct
The issue struct contains the parameters described in the following table.
| Parameter | Type | Description |
|---|---|---|
| var severity | issue_severity | The severity of the issue. |
| var category | issue_category | The category of the issue. |
| var kind | issue_kind | The kind of the issue. |
| var details | string | The details provided for the issue. |
| var normalized_details | string | The normalized details provided for the issue. |
| var time | time | The time that the issue was raised. |
| var additional_data | sdl_struct | Additional data relevant for the issue (type depends on issue_kind, might be null). |
| var modification_reason | string | Reason for updates to the issue. |
sut_issue()
Use this method to create a new issue with its category set to sut.
sut_issue([severity:] <issue_severity>, [kind:] <issue_kind>, [details:] <string>, [[normalized_details:] <normalized details string>])
severity: issue_severity-
Sets the severity for the new issue.
kind: issue_kind-
Sets the kind for the new issue.
details: string-
Sets the details for the new issue.
normalized_details: string-
(Optional) Sets the normalized details for the new issue. If not provided or an empty string is provided, normalized_details is automatically created based on details.
sut_error() issue
Use this method to create an error issue with its category set to sut and it severity set to error.
sur_error([kind:] <issue kind>, [details:] <details string>, [[normalized details:]<normalized details string>])
kind: issue_kind-
Sets the kind for the new issue.
details: string-
Sets the details for the new issue.
normalized_details: string-
(Optional) Sets the normalized details for the new issue. If not provided or an empty string is provided, normalized_details is automatically created based on details.
sut_warning()
Use this method to create an error issue with its category set to sut and it severity set to warning.
sut_warning([kind:] <issue_kind>, [details:] <string>, [[normalized_details:] <normalized details string>])
kind: issue_kind-
Sets the kind for the new issue.
details: string-
Sets the details for the new issue.
normalized_details: string-
(Optional) Sets the normalized details for the new issue. If not provided or an empty string is provided, normalized_details is automatically created based on details.
scenario_completion_error()
Use this method to create a new issue with its category set to scenario_completion and its severity set to error.
scenario_completion_error([kind:] <issue_kind>, [details:] <string>, [[normalized_details:] <normalized details string>)
kind: issue_kind-
Sets the kind for the new issue.
details: string-
Sets the details for the new issue.
normalized_details: string-
(Optional) Sets the normalized details for the new issue. If not provided or an empty string is provided, normalized_details will is created based on details.
scenario_completion_warning()
Use this method to create a new issue with its category set to scenario_completion and its severity set to warning.
scenario_completion_warning([kind:] <issue_kind>, [details:] <string>, [[normalized_details:] <normalized details string>)
kind: issue_kind-
Sets the kind for the new issue.
details: string-
Sets the details for the new issue.
normalized_details: string-
(Optional) Sets the normalized details for the new issue. If not provided or an empty string is provided, normalized_details is automatically created based on details.
other_issue()
Use this method to create a new issue with its category set to other.
other_issue([severity:] <issue_severity>, [kind:] <issue_kind>, [details:] <string>, [[normalized_details:] <normalized details string>)
severity: issue_severity-
Sets the severity for the new issue.
kind: issue_kind-
Sets the kind for the new issue.
details: string-
Sets the details for the new issue.
normalized_details: string-
(Optional) Sets the normalized details for the new issue. If not provided or an empty string is provided, normalized_details is automatically created based on details.
other_error()
Use this method to create a new issue with its category set to other and its severity set to error.
other_error([kind:] <issue_kind>, [details:] string, [[normalized_details:] <normalized details string>)
kind: issue_kind-
Sets the kind for the new issue.
details: string-
Sets the details for the new issue.
normalized_details: string-
(Optional) Sets the normalized details for the new issue. If not provided or an empty string is provided, normalized_details is automatically created based on details.
other_warning()
Use this method to create a new issue with its category set to other and its severity set to warning.
other_warning([kind:] <issue_kind>, [details:] <string>, [[normalized_details:] <normalized details string> )
kind: issue_kind-
Sets the kind for the new issue.
details: string-
Sets the details for the new issue.
normalized_details: string-
(Optional) Sets the normalized details for the new issue. If not provided or an empty string is provided, normalized_details is automatically created based on details.
raise_issue()
Use this method to create a new issue with its category set to other and its severity set to warning.
raise_issue([severity:] <issue_severity>, [kind:] <issue_kind>, [category:] <issue_category>, [details:] <string>, [[normalized_details:] <normalized details string>], [[additional_data:] <sdl_struct>])
kind: issue_kind-
Sets the kind for the new issue.
details: string-
Sets the details for the new issue.
normalized_details: string-
(Optional) Sets the normalized details for the new issue. If not provided or an empty string is provided, normalized_details is automatically created based on details.
additional_data: sdl_struct-
(Optional) Sets the additional data. Sets additional_data field under issue.
sut_error_if()
Use this method to create a new issue conditionally with its category set to sut and its severity set to error.
sut_error_if([val:] <bool>, [kind:] <issue_kind>, [msg:] <string>)
val: bool-
Issue is created if val is True.
kind: issue_kind-
Sets the kind for the new issue.
msg: string-
Sets the issue details.
fail_if()
Use this method to create a new issue conditionally with its category set to scenario_completion and its severity set to error.
fail_if([val:] <bool>, [kind:] <issue_kind>, [msg:] <string>)
val: bool-
The issue is created if val is True.
kind: issue_kind-
Sets the kind for the new issue.
msg: string-
Sets the Issue details.
assertion_error()
Use this method to create a new issue with its category set to other, its severity set to error, and its kind set to assertion.
assertion_error([msg:] <string>)
msg: string-
Sets the Issue details.
assertion_error_if()
Use this method to create a new issue conditionally with its category set to other, its severity set to error, and its kind set to assertion.
assertion_error_if([val:] <bool>, [msg:] <string>)
val: bool-
The issue is created if val is True.
msg: string-
Sets the Issue details.
Resolving common issues
This section describes common issues and suggests ways to resolve them.
See Also:
load_failed
Foretify failed to load all the files required for a test, usually because the pathname of a file was not correct or incomplete. Check the path and filename and correct it.
These are the search rules used by Foretify when loading files:
-
If a full path is specified (for example, /x/y/my_file), then use that full path.
The file name as specified is looked up first, and if not found, the name with the extension '.osc' is looked for.
-
Else look in the directory where the loading file resides.
-
Else look in the current directory.
-
Else look in directories specified by the OSC_PATH environment variable.
-
Else issue a load_failed error.
Nested lists with non-deterministic sizes
When the OSC code has nested lists, i.e., a list of lists or a list of structs where a struct has a field of list type, only one level of the list can have a non-deterministic size.
Non-deterministic size refers to a list whose length is not fixed and can vary, such as lists defined with constraints on their size. For example, a list with keep(it.size() in [3..6]) would have a non-deterministic size because its length can be anywhere from 3 to 6. While it is allowed to have such lists at one level, having multiple levels with non-deterministic sizes simultaneously is not supported. Only either the outer or inner list can be non-deterministic, but not both at the same time.
Example unsupported case
In this example, both outer list f and inner list l have non-deterministic sizes.
struct foo:
l: list of int with:
keep(it.size() in [5..8])
extend top.main:
f: list of foo with:
keep(it.size() in [2..10])
Example supported case
In this example, the inner list l has a deterministic size and the outer list f has a non-deterministic size, making the configuration supported.
struct foo:
l: list of int with:
keep(it.size() == 5)
extend top.main:
f: list of foo with:
keep(it.size() in [2..10])
Constraint s == null under non-deterministic condition
Constraints of the kind s == null, where s is a struct, are only supported if they can be statically resolved. Cases where s can be randomly set to either null or non-null are unsupported.
Example unsupported cases
In this example, s is null if the flag is randomized to true, making it non-deterministic.
struct S:
. . .
extend top.main:
s: S
keep(s == null)
See complete example here.
In the following example, depending on the randomization of b.f, s is either null (if b.f == true) or non-null (if b.f == false).
struct base:
f: bool
s: S
struct derived1 inherits base(f == true):
keep(s == null)
struct derived2 inherits base(f == false)
extend top.main:
b: base
See complete example here.
Example supported cases
The following cases are supported because b.f can be statically resolved:
b: derived1 # b.f is statically resolved to true
b: base with:
keep(it.f == true) # b.f is statically resolved to true
See complete example here.
s == null error message
In case of an unsupported s == null constraint, the following error message is issued:
Constraining struct to be either null or non-null under a non deterministic condition is not supported yet, <source reference>
compilation_error
Foretify issues a compilation error if any OSC2 file contains code that does not meet OSC2 coding requirements. The compiler specifies the filename and line number where the error was encountered as part of the error message. Check the OSC2 file at that given position for syntax errors.
unfeasible_plan
At the beginning of the execution phase, Foretify performs compliance checks to make sure that the values generated for this seed satisfy all modifiers. If at least one of the generated values does not satisfy a modifier, an error of type unfeasible_plan is raised and the generated value and unsatisfied modifier are displayed:
[INFO] [MAIN] *** Error: Unfeasible plan (unfeasible_plan): An unfeasible plan has been created and can not be executed:
main.label(sut).ego_drive_initial_phase - <truck@30 (top.sut.car) red uid:20> lane is: 2, but not in the range requested: [3..3]
To resolve this issue temporarily, you can re-run the test with a different seed, using the --seed invocation option for Foretify. Because the values generated for the test vary according to the seed, it is usually possible to find a seed where the generated values satisfy all compliance checks. However, this is not a complete solution.
For a more lasting solution, identify and fix the problematic coding style that is causing the plan to be unfeasible:
-
Load the test into Foretify with the --lint error invocation option.
See Using the Foretify Linter for more information.
-
In the output from the linter, identify all messages related to "modifier", for example:
Foretify linter outputWARN lint_absolute_speed Absolute speed in 'label(cut_out.cut_out_vehicle)' drive scenario invocation -
In the user documentation, search for the linter rule by name.
For example, the name of the linter rule identified above is lint_absolute_speed.
incomplete_scenario
Foretify issues an incomplete_scenario error if a test condition is not met. For a very simple example, assume that car1 is required to drive within a specified speed range:
car1.drive() with:
speed([20..50]kph, at: end)
See complete example.
During the generation phase, Foretify chooses a specific value for car1's speed from the given range. During the execution phase of this drive, Foretify checks the actual speed, as reported by the simulator, and allows finishing the drive only when actual speed is within the range. If for some reason, the required speed is not reached within the planned time, Foretify might extend the drive. If car1 still fails to reach the required speed, Foretify issues an incomplete scenario error.
An incomplete scenario error might be raised in two cases:
-
The test reached the maximum allowed duration without completing. See Avoiding tests that are too long.
In this case, the details of the issue will include a list of running scenarios and a list of unsatisfied conditions. For example:
Incomplete scenario message for reaching max allowed duration[ERROR] [10.000] [MAIN] *** Error: Incomplete scenario (incomplete_scenario): Test reached max allowed duration (10second) set by config.test.max_test_time during following active scenarios: main.label(sut) at line 17 in /ftx/test/max_duration_incomplete.osc Can't transition to next scenario because of: main.label(sut) - <sut_vehicle@1 (top.sut.car) red uid:20> speed: 0mps, but is not in requested range: [13.89..13.89]mps (equivalent to [13.33..14.44]mps +/- 0.56mps tolerance) -
The scenario conditions don't allow the test execution to continue or to end (or switch to the next scenario). For a simple example, assume we are running the following code:
OSC2 code: incomplete scenario examplecar1.drive() with: speed([20..50]kph, at: end) along(some_road)In this example, the scenario requires car1 to reach a speed between 20kph and 50kph, but the whole drive is required to happen on some_road. If car1 fails to reach the required speed and leaves some_road, an incomplete scenario error occurs. The incomplete scenario issue in such a case will include the following details: - The list of running scenarios - The list of conditions preventing the test from continuing to execute the current scenarios - The list of conditions preventing the test from ending the current scenario or switching to the next scenario
Following is an example of an incomplete scenario message.
Incomplete scenario message*** Error: Incomplete scenario (incomplete_scenario): Active scenarios: main.label(sut).sut_cut_out.label(cut_out.lead_vehicle) at line 15 in /ftx/packages/base_scenarios/scenarios/ego_cut_out/sut_cut_out/sut_cut_out_imp.osc main.label(sut).sut_cut_out.label(cut_out.sut) at line 13 in /ftx/packages/base_scenarios/scenarios/ego_cut_out/sut_cut_out/sut_cut_out_imp.osc Can't continue current scenario because of: main.label(sut).sut_cut_out.label(cut_out.duration) - reached max allowed duration of 4s Can't transition to next scenario because of: main.label(sut).sut_cut_out.label(cut_out.sut) - <sut_vehicle@22 (top.sut.car) red uid:44> changed 0 lanes but not at the requested range: [1..1]
There are various reasons why a scenario may fail to complete:
-
Problematic coding style
You can use the Foretify linter to identify and eliminate coding style issues that prevent Foretify from creating a plan that can be executed to completion. See unfeasible_plan for instructions on how to use the linter to do this.
-
The SUT behaves in a way that Foretify does not expect.
Foretify needs a way to predict the behavior of the SUT. The best way to do this is to set a constraint on sut.car that describes the expected behavior. For example, set a keep_lane() or change_lane() to describe the expected behavior of the SUT when approaching a slowing lead vehicle in the same lane.
-
Another actor behaved in a way that Foretify does not expect.
Constraints on the SUT are required if the NPC is expected to move relatively to the SUT. In the following example, the speed() modifier on sut.car.drive() is required:
OSC2 code: constrain NPC to move relatively to SUTdo parallel(overlap:equal): sut.car.drive() with: speed([35..50]kph) # this modifier is required because of # the relative constraint on car1's speed car1.drive() with: # Faster than sut.car by [1..5]kph speed([1..5]kph, faster_than: sut.car)See complete example.
simulation
A common simulation issue can arise if you forget to configure the simulator or other execution platform. To fix this issue, you can import the configuration file from a simulator.
extend test_config:
set map = "$FTX_PACKAGES/maps/hooder.xodr"
extend top.main:
do serial():
sut.car.drive() with:
speed([50..100]kph, at: all)
[INFO] Compilation done, 0 errors and 0 warnings reported
...
Starting the test ...
Running the test ...
*** Error: Other error (simulation): Simulator is not defined -
stopping the run at line 121 in @sdl_issue_imp
...
See complete example.
collisions (collision_in_hybrid_control, collision_in_sut_control, collision_in_manual_control)
Foretify defines three types of collisions:
-
collision_in_hybrid_control: Reports a collision while a SUT was engaged driving the vehicle in parallel to the Foretellix Human Driver Model (usually this is teh case for ADAS type SUTs).
-
collision_in_sut_control: Reports a collision while the SUT was in full control of the vehicle.
-
collision_in_manual_control: Reports a collision while the SUT was not in any control of the vehicle, instead the vehicle was fully controlled by the Foretellix Human Driver Model, this could mean it is a NPC vehicle or an ADAS type SUT with deactiovated ADAS system.
These kinds of collisions are defined separately because you might want to handle them differently. While if you want to analyze why the SUT caused a collision, you might want to downgrade the severity of the other issues temporarily, in most cases collision_in_manual_control will need investigation by Foretellix.
Collision error messages
Foretify provides additional information to assist in debugging, when it is available. These messages vary depending on whether the SUT or an NPC caused the collision and also on the type of driver that controlled the colliding vehicle:
- The SUT_driver (the SUT) controls both lateral and longitudinal movement of the sut.car for a Level 3 or 4 SUT.
- The hlm_driver (Foretellix SUT Model) controls both lateral and longitudinal movement of a High-Level Model (HLM) of the sut.car.
- The ftx_driver (the Foretellix Human Driver Model) controls both lateral and longitudinal movement of the sut.car or of an NPC that takes an active role in the scenario.
- For the hybrid_driver, the ftx_driver controls either the lateral or the longitudinal movement of the sut.car and the SUT_driver controls the other.
- The external_driver (typically the simulator) controls both the lateral and longitudinal movement of an NPC.
Depending on whether the SUT or the NPC caused the collision and on the type of driver, additional information may be provided about the possible root cause in the following format:
SUT | NPC: <driver> is in control: unexpected behavior with collision.
Potential root cause: <cause>.
In the case of SUT: SUT_driver or NPC: external_driver a possible root cause is collision on creation. This kind of collision happens when the sut.car or the NPC is created in close proximity to another actor.
Possible root causes of collision for the SUT: hybrid_driver, the SUT: ftx_driver and the NPC: ftx_driver include:
- collision_avoidance_disabled: by default, all actors avoid collisions, but this function can be disabled with the avoid_collisions() movement modifier or by setting the behavior_manager_params.post_AEB_override_behavior.enable parameter for a vehicle to true (the default).
- environment_object_movement: an object in the environment moved, perhaps in an unexpected manner.
- unavoidable_collision: given the constraints on the movements of various actors in the scenario, the collision could not be avoided.
For SUT: hybrid_driver, additional information about the collision may be provided from the simulator, such as whether the vehicles were in the same or different lanes, whether an obstacle was recognized or not, and so on.
Failed to create a plan
For some tests, Foretify may sometimes fail to create a plan for executing the scenario. In that case, you will see an error like this:
[ERROR] [0.000] [MAIN] InfiniGen failed to create a plan for the test. For contradiction check, run foretify with '--set config.gen.contradiction_check=true'.
When running foretify with --set config.gen.contradiction_check=true, the contradiction report is:
[ERROR] [0.000] [MAIN] Contradiction was found in the following set of constraints:
< set of constraints that caused a contradiction>
The general recommendation is to not run with --set config.gen.contradiction_check by default but only for the purpose of contradiction debugging when you receive the "Failed to create a plan" error message.
In order to resolve this issue, you can do either of the following:
- Review the contradictions provided and adapt your scenario or parameters accordingly to not run into the same contradiction.
- Run the test with a different seed.
For more details about contradictions, see Contradiction detection.
exception and assertion issues
You should report any exception and assertion issues arising from Foretify itself to Foretellix.
Loaded map is too large
Maps are usually the most memory-consuming element Foretify generator needs to deal with. For this reason, and to avoid out-of-memory issues in the solver, Foretify limits the size of the memory that can be loaded by the tool. Since this limitation is dependent on the amount of the available memory on the user machine, this memory limit is configurable via the following setting:
``` title="OSC2 code: setting the map memory threshold
extend gen_config:
set map_memory_threshold_MB = The default value of this setting is 4GB.
Note that the memory size is not the actual physical size of the memory file, but rather an amount of memory that will be required by the solver (which often is also scenario dependent), so it’s not easy to estimate in advance whether the map is below or above the given threshold.
In case Foretify identifies that the loaded map exceeds the given threshold, the following error message will be issued:
``` title="Foretify output: loaded map is too large error"
[MODELER] Estimated memory for handling the map during generation is {} MB, which exceeds the threshold of {} MB. Please consider using smaller ROIs or a different map. To redefine the threshold, run foretify with '--set config.gen.map_memory_threshold_MB=new_threshold.
If you get such a memory there are several ways to deal with it:
- If you have a sufficient amount of memory on your machine, just try to increase the threshold
- Otherwise, try to reduce your map by taking only a part of it that is of interest for this particular scenario (using ROI (region of interest))
Other ways to resolve an issue
In addition to modifying your OSC2 code or SUT behavior, you can resolve an issue by modifying its attributes, such as severity or category.
Foretify provides various builtin methods related to issue resolution, depending on whether you want to:
- Modify a defined issue
- Modify an issue for a selected scenario
- Modify an issue across a test suite, regardless of whether it is defined for all tests
Modifying a defined issue
If an issue is defined for all tests in a group or test suite, you can modify its severity level or category when it is created, using the event top.handle_issue and the external method <current-issue>.modify() or <current-issue>.modify_category(). The current issue is accessible via top.issues.curr.
The modify() method changes the severity level of an issue.
modify(kind: <issue_kind>, new_severity: <issue_severity>, reason: <string>)
The example below extends top.main to downgrade the severity of all custom_kind issues to info.
extend top.main:
on @top.new_issue:
issues.curr.modify(custom_kind, info,
"downgrading custom kind severity to ‘info’")
See complete example.
The modify_category() method changes the category of an issue.
modify_category(kind: <issue_kind>, new_category: <issue_category>, reason: <string>)
If the issue's kind is <issue_kind>, this method replaces its category with <issue_category> and replaces the currently defined details with <string>.
This example extends top.main to change the category of all sensor_failure issues to other.
extend top.main:
on @top.new_issue:
issues.curr.modify_category(sensor_failure, other,
"This issue is not due to a sensor failure.")
See complete example.
Modifying an issue for a selected scenario
There may be cases where you want to change the severity or category of an issue only within a specific scenario.
Foretify provides the top.new_issue event, which is emitted when an issue is created, and several external methods of any_scenario that help you with issue resolution within a specific. These methods are:
any_scenario.is_running(): bool
any_scenario.is_done(): bool
any_scenario.completion_error(kind: <issue_kind>, details: <string>)
any_scenario.completion_warning(kind: <issue_kind>, details: <string>)
This test changes the severity level of a sensor failure from error to warning if it occurs in top.main.scenario1().
extend top.main:
on @new_issue if scenario1.is_running():
issues.curr.modify(sensor_failure, warning,
"In top.foo, if scenario1 is running, all sensor failures are warnings")
See complete example.
This test overrides the severity of all sensor failures that occur during the run.
extend top.main:
def ignore_sensor_failures(kind: issue_kind) is:
if kind == sensor_failure:
issues.curr.modify(sensor_failure, ignore,
"In top.main, all sensor failures are ignored")
on @new_issue:
ignore_sensor_failures(issues.curr.kind)
See complete example.
Modifying issues for a test suite
If you use issues.curr.modify() or issues.curr.modify_category() to modify an issue, and that issue is not defined by an extension to issue_kind, you see a compilation error.
When running a test suite, you can change the severity, category or kind of an issue for the entire suite, even if that issue_kind extension is not loaded by all the tests. This is useful when you want to run different versions of the SUT with the same tests and need to adapt verdict analysis accordingly.
# syntax
modify_severity_by_name(name: <string>, new_severity: <issue_severity>, reason: <string>)
# example
on @top.new_issue:
issues.curr.modify_severity_by_name("sensor_failure", warning, "Moving sensor failure check to warning")
See complete example.
# syntax
modify_category_by_name(name: <string>, new_category: <issue_category>, reason: <string>)
# example
on @top.new_issue:
issues.curr.modify_category_by_name("sensor_failure", sutv2, "Moving sensor failure check to 'sutv2'")
See complete example.
# syntax
modify_kind_by_name(name: <string>, new_kind: <issue_kind>, reason: <string>)
# example
on @top.new_issue:
call issues.curr.modify_kind_by_name("sensor_failure", other, "Changing sensor failure issue kind to 'other'")
See complete example.
Custom resolution logic
You can also call an external method that implements custom resolution logic. For example:
extend top.main:
on @new_issue:
if (issues.curr.kind == overspeed and sut.car.active_speed_sensor == radar and sut.car.state.speed < 60kph):
issues.curr.kind = sensor_failure
issues.curr.severity = warning
See complete example.
Limiting the number of duplicate issues reported
Foretify limits the number of duplicate issues printed to the log and counted in metrics for issues with severity error_continue and below.
If multiple issues with severity error_continue and below have the same category, kind, severity and normalized message, by default only the first five occurrences are printed to the log and counted in metrics.
You can change this default by setting issues.max_duplicate_issues. For example:
extend issues:
set max_duplicate_issues = 3
See complete example.
Adding issue metrics to the VPlan
You can add the issue metrics created by Foretify to a VPlan in order to include these issues in the overall grade of a test suite.
The following issue metrics are recorded under top.info in group main_issue, once at end of run:
- category
- severity
- kind
- details
- time - the time in which the main issue was reported
- full_message - category + severity + kind + details + modification reasons
- normalized_details
- modification_reason
- result - category + severity
- kind__severity - kind crossed with severity
- category__severity - category crossed with severity
Note
To enable clustering by details, a normalized_details string metric is derived from the details string . The normalization removes numbers and text inside angled brackets (‘<>’).
The following issue metrics are recorded under top.info in group issue, for every new issue:
- i_category
- i_severity
- i_kind
- i_details
- i_time
- i_full_message
- i_normalized_details
- i_modification_reason
- i_kind__i_severity - kind crossed with severity
- i_category__i_severity - category crossed with severity