Skip to content

Defining coverage metrics

Coverage metrics capture the parameters that define key aspects of a test. For example, you might want to collect the following coverage metrics for a scenario that places an object on the route of the Ego:

  • What kind of objects did the Ego encounter on its route?
  • Were the objects positioned to the left or right of the Ego?
  • How far from the Ego were the objects at the start of the movement?

The OSC2 cover() construct defines coverage metrics. For example, you can specify the type of object the Ego encounters (such as VRU, pedestrian, or vehicle) and set a target number of encounters for each object type during the regression.

Coverage metrics defined in the top.info struct are collected globally across all runs of all tests. Coverage metrics can also be defined locally for each scenario or for a group of related scenarios.

After creating coverage metrics and running your test, you can upload the results to Foretify Manager to view the metrics. See Analyzing metrics to learn how to use a Verification plan (VPlan) and work with the metrics.

Defining coverage files and struct files

The coverage metrics are defined in a cover.osc file.

Example coverage file

    ...

    # Road coverages
    # Tracking various type of road
    ego_road_element: sut.car.track_road_element(one_way_road, trigger: start)
    road_tracking: sut.car.track_road_element(road_element, trigger: all)
    junction_road_tracking: sut.car.track_road_element(internal_road, trigger: start)
    road_with_sign_tracking: sut.car.track_road_element(road_with_sign, trigger: start)
    highway_road_tracking: sut.car.track_road_element(highway_base, trigger: start)
    merging_lane_tracking: sut.car.track_road_element(road_with_merging_lane, trigger: start)
    curved_road_tracking: sut.car.track_road_element(road_curvature, trigger: start)
    speed_limit_road_element: sut.car.track_road_element(speed_limit_section, trigger:start)

    event ego_drove_on_curved_road_element is @curved_road_tracking_event if(
        curved_road_tracking_data.direction != neither)

    cover(curve_road_direction,
        expression: curved_road_tracking_data.direction,
        text: "Direction of the road curvature",
        event: ego_drove_on_curved_road_element)


    # Method to track type of road
    var changed_road_type:bool
    def ego_driving_on_road_type() is:
        var prev_road_type := main_road_type
        if junction_road_bool == true:
            set main_road_type = junction
        elif merging_lane_road_bool == true:
            set main_road_type = merging_lane
        elif highway_road_bool == true:
            set main_road_type = highway
        else:
            set main_road_type = off_highway
        if main_road_type != prev_road_type:
            changed_road_type = true
        else:
            changed_road_type = false

    # Road type cover
    event road_tracking_changed is @road_tracking_event if changed_road_type == true
    cover(main_road_type,
        expression: main_road_type,
        text: "Type of road in the scenario on which the Ego was driving",
        event: road_tracking_changed,ignore: main_road_type == none)

    # Coverage of Curve road
    ## Minimum road radius
    cover(min_road_element_radius,
        expression: min_road_element_radius,
        unit_string: ftlx_lib_base_config.default_curvature_unit,
        expression: curved_road_tracking_data.min_radius,
        buckets: [0, 100, 150, 200, 300, 400, 600, 800, 1000,
            1200, 1400, 1600, 1800, 2000, 2400, 2800,
            3200, 3600, 4000, 5000, 6000, 8000, 10000, 20000] ,
        text: "Minimum radius of the road",
        event: ego_drove_on_curved_road_element)

    # Road curvature
    var flag_sut_is_not_null: bool = false
    event record_curve_radius_event is @finish if flag_sut_is_not_null

    avg_curve_radius_collect: statistics(sample_type: length, measurement: average)
    on @top.clk:
        if sut.car.state != null:
            flag_sut_is_not_null = true
            avg_curve_radius_collect.add_sample(\
                math.abs(sut.car.get_road_radius())
            )
    cover(avg_curve_radius,expression: avg_curve_radius_collect.compute(),
        unit_string: ftlx_lib_base_config.default_distance_unit,
        range: [0..4900],
        every: 150,
        text: "Average curve radius throughout the scenario",
        event: record_curve_radius_event)

    # Speed of Ego coverages
    ## Ego speed on curved road
    var ego_speed_on_curve_road_element := sample(sut.car.state.speed,
        @ego_drove_on_curved_road_element)
    cover(ego_speed_on_curve_road_element,
        expression: ego_speed_on_curve_road_element,
        unit_string: ftlx_lib_base_config.default_speed_unit,
        range: ftlx_lib_base_config.default_ego_speed_cover_range,
        every: ftlx_lib_base_config.default_ego_speed_cover_bin,
        text: "Speed of the Ego on the curved road",
        event: ego_drove_on_curved_road_element)

    # Input coverage
    cover(gen_ego_speed_at_start,
        expression: gen_ego_speed_at_start,
        unit_string: ftlx_lib_base_config.default_speed_unit,
        range: ftlx_lib_base_config.default_ego_speed_cover_range,
        every: ftlx_lib_base_config.default_ego_speed_cover_bin,
        text: "Generated speed of the Ego at the start of the scenario",
        event: start)

    # Actual coverage
    cover(ego_speed_at_start,
        expression: ego_speed_at_start,
        unit_string: ftlx_lib_base_config.default_speed_unit,
        range: ftlx_lib_base_config.default_ego_speed_cover_range,
        every: ftlx_lib_base_config.default_ego_speed_cover_bin,
        text: "Actual speed of the Ego at the start of the scenario",
        event: start)

Ranges and bins are defined as consts and grouped in a struct to centralize configuration for a library, for example, the following snippet from the struct ftlx_lib_base_config.osc file:

struct ftlx_lib_base_config:

   # General units
    # Speed units
    const default_speed_unit: string = "kph"
    const default_speed_limit_unit: string = "kph"
    const default_lat_speed_unit: string = "mps"
    const default_angular_speed_unit: string = "radps"

   # Ego related consts
    # Ego speed constants
    const default_ego_speed_range: range of speed = [0..150]kph
    const default_ego_speed_cover_range: range of int = [0..150]
    const default_ego_speed_cover_bin: uint = 10

Defining simple coverage

In the following code snippet, the cover function creates a coverage point for the Ego's input speed at the start of each scenario. This is tracked under the variable gen_ego_speed_at_start.

# Input coverage
cover(gen_ego_speed_at_start,
    expression: gen_ego_speed_at_start,
    unit_string: "kph",
    range: [0..20],
    every: 2,
    text: "Input speed of the Ego",
    event: start)

The cover() function tracks sampled speeds across multiple tests using the defined parameters and configuration values, creating a map of covered and uncovered speed bins.

Defining coverage for a sampled parameter

In the following code snippet, at the @start event (the beginning of the scenario), the value of sut.car.state.speed (the Ego’s actual speed) is captured, and the value is recorded for coverage as the ego_speed_at_start variable. In other words, it records the Ego’s speed at the start of the scenario for coverage analysis and reporting.

on @start: ego_speed_at_start = sut.car.state.speed
 cover(ego_speed_at_start,
    expression: ego_speed_at_start,
    unit_string: "kph",
    range: [0..30],
    every: 2,
    text: "Speed of the Ego at the start of the scenario",
    event: start)

Defining coverage for sampled parameter on an event

In the following OSC2 code snippet, when the Ego is driving on a curved road (event) at a speed between 0 and 150 kilometers per hour, a sample of of the speed is taken, labeled, and stored in the variable ego_speed_on_curve_road_element for coverage and analysis. 

## Ego speed on curved road
    event ego_drove_on_curved_road_element is @curved_road_tracking_event if(
        curved_road_tracking_data.direction != neither)
    var ego_speed_on_curve_road_element := sample(sut.car.state.speed, 
        @ego_drove_on_curved_road_element)
    cover(ego_speed_on_curve_road_element, 
        expression: ego_speed_on_curve_road_element,
        unit_string: ftlx_lib_base_config.default_speed_unit,
        range: ftlx_lib_base_config.default_ego_speed_cover_range, 
        every: ftlx_lib_base_config.default_ego_speed_cover_bin,
        text: "Speed of the Ego on the curved road",
        event: ego_drove_on_curved_road_element)

Here's a breakdown of each line:

Parameter Description
event ego_drove_on_curved_road_element Event at which to take a sample speed
var ego_speed_on_curve_road_element Variable to hold the sampled speed
sample(sut.car.state.speed,
@ego_drove_on_curved_road_element)		 Specifies to take a sample, and where to assign the value
cover(ego_speed_on_curve_road_element) Defines the coverage metric using the attributes unit_string, range, every, text, and event to specify units of measurement, range of values, bucketing, labeling, and collection timing.