370. Introduction to Global Evaluators

Global evaluators are part of the Evaluation V-Suite Library. Events of interest trigger them and can identify critical or unsafe behavior or add context to a situation. These evaluators are enabled by default for the generative runs and evaluation pipeline across all scenarios.

Global Evaluators are organized in two categories:

• Global Watchers focus on observable conditions by adding contextual intervals, helping with triaging and reducing the number of needed KPIs in the evaluation scenarios. Examples include the curvature of a road, the current speed limit, the type of road, or the weather.

• Global Checkers focus on observable Ego behavior, primarily identifying situations that indicate unsafe or unintended driving behavior. Examples include ride comfort events, driving too close to objects, collisions, and lane departure situations.

Global evaluators create intervals and additional data in the form of KPIs, coverage items, and contextual data in addition to the evaluation scenario matches. The combination of scenarios and global evaluators enables the user to identify relevant situations and analyze the verdict of such situations based on the additional context provided by the evaluators.

These intervals are created based on specified triggering conditions or by analyzing the current driving environment, regardless of whether real-world or simulation data is injected into the Foretellix toolchain. Depending on the type of event and evaluator, the intervals can vary in length from single-frame events to longer intervals.

370.1 Parameter Customization for Global Evaluators

Global evaluators incorporate user-configurable parameters to control key aspects of their behavior. Default values for these parameters are defined in the config file located at $FTX/tool/user_interface/domain_model/checks/ftlx_global_eval_config_top.osc. Users can override these default values to suit their specific requirements. Each global evaluator uses this configuration during setup, eliminating the need for code modifications.

To customize these parameters, the user should create a custom config file by extending the default ftlx_global_eval_config struct and setting the required values. Below is an excerpt of the default config file, located at $FTX/tool/user_interface/domain_model/checks/ftlx_global_eval_config_top.osc:

struct ftlx_global_eval_config:
    # General units
    const default_speed_unit: string = "kph"

    const ego_static_object_too_close_checker_config = static_object_too_close_checker_config = new
    # Default distance threshold to trigger the static_object_too_close checker
    set ego_static_object_too_close_checker_config.default_obj_too_close_distance_threshold = 0.5m

The customized file may look like this:

# New file for user configuration
extend ftlx_global_eval_config:
    set default_speed_unit = "mph"

    # Disable a specific checker
    set ego_kinematic_spec_checker_config.enabled = false

    # Adjust other parameters for enabled checkers
    set ego_static_object_too_close_checker_config.default_obj_too_close_distance_threshold = 1m

370.2 Global Checkers

370.3 Global Watchers