Skip to content

114. Smart Replay accuracy metrics

Traces and Key Performance Indicators (KPIs) for Smart Replay are used to assess how accurately each vehicle, including the Ego, follows its original trajectory from the source run.

114.1 Smart Replay mode metrics

Smart Replay modes affect how the metrics should be interpreted.

  • Exact mode metrics verify the accuracy of the replay against the source run and validate that all vehicles, including the Ego during warmup, adhere to their expected paths.

  • Behavioral mode metrics verify how closely vehicles replicate the source run. Minor deviations are expected and considered valid, particularly when they result from interaction with a changing Ego trajectory.

114.2 Trace Metrics

114.2.1 Distance-based comparison

All accuracy metrics are based by measuring the actual position of each vehicle during the test to its recorded position from the source run at each timestamp.

  • The distance used is the Cartesian distance in meters.

  • If a vehicle's position is unavailable at a timestamp, the value of those timestamps will not be recorded.

Figure 1: Unrecorded timestamps

For each vehicle, a trace named dist_from_rec_pos is displayed in the Traces view under its replay scenario.

Figure 2: Trace metrics

114.3 KPIs

For each actor, the following KPIs are calculated:

  • Maximum distance
  • Average distance
  • Standard deviation

These KPIs can be viewed in the VPlan under any of the replay scenarios.

Figure 3: Actor KPIs

114.4 Scenario-level metrics

Scenario-level metrics aggregate the local metrics of all actors to provide a high-level summary of the scenario's overall performance.

Figure 4: VPlan metrics

The following scenario-level metrics are recorded:

  • global_max_sd_distance_error_npc

    The maximum standard deviation of the distance error for NPCs in the Smart Replay scenario. This metric summarizes how much the replayed trajectories deviate, in terms of variability, from the reference trajectories, considering all samples and all actors.

  • global_max_sd_distance_error

    The maximum standard deviation of distance error for any actor in the scenario replay is measured across all timestamps. This metric quantifies the deviation of the most inconsistent actor's replayed trajectory from its original path. It serves as an indicator of the worst-case spatial drift encountered for any actor in a run.

  • global_max_distance_error_npc

    The maximum distance deviation, across the entire scenario, between the position of any NPC during a Smart Replay run and its corresponding position in the original recorded trajectory at each timestamp. This metric is particularly useful in Behavioral Mode, where NPCs adapt to changes (such as the Ego acting differently) and cannot always follow their exact trajectories, though high accuracy is still desired.

  • global_avg_distance_error

    The mean of the average distance errors across all actors in the scenario. This provides a single value representing the overall mean deviation from the expected positions for all vehicles during the scenario.

114.4.1 Aggregation metrics

The following metrics are calculated for all actors based on various measures. For instance, a metric counts the number of actors whose distance error falls within a range, such as [0..0.25].

  • Maximum: The highest value observed across all actors, along with its originating actor. For example, the distance error for an actor that falls within the [0..0.25] range is included in this bucket hits.
  • Average: The mean value calculated across all actors. For example, the mean value of a distance error for an actor that falls within the [0..0.25] range is included in this bucket hits.
  • Standard deviation: The highest standard deviation value observed across all actors, along with its originating actor. For example, the standard deviation value of a distance error for an actor that falls within the [0..0.25] range is included in this bucket hits.
Figure 5: Aggregation metrics