109. Ensuring accurate actor representation in Smart Replay

To ensure accurate replay of real-world scenarios in Smart Replay, it's important to preserve the characteristics of different actor types. For example, a truck in the real world should not be replayed as a sedan or a motorcycle. For more information refer to Passing data from the real world to Smart Replay.

109.1 Technical limitation

Smart Replay has a feature that enables persisting actor data in Foretify runs to be used later in Smart Replay scenarios. This ensures that the correct actor types are replayed with the required fields, maintaining consistency with the original run.

The stored data consists of static actor data that defines the characteristics of the actors, such as size, color, and category for vehicles, as well as kind for stationary objects, and other relevant attributes.

This works well in Generation V-Suite scenarios because this data is static, it is generated once per actor, and remains constant throughout the run.

However, it fails when ingested object lists are compared against specified evalution scenarios because some actor data cannot be updated during execution for the following reasons:

• In ingested runs, actors are reused and therefore the data must change over time. However, some actor data is changeable and is updated during the run, in which case it can work. This is the case for the bounding box of an actor.
• Some actor data is stored in the actor fields that are used for conditional inheritance. In this case, even when the data is known, it cannot be updated for an actor after it is first created. As a result if the data is persisted, it will only be correct some of the time, making it unreliable. Examples include the category field for vehicles and the kind field for plain objects.

109.2 Solution

The solution requires writing OSC code and may require C++ extensions. Throughout this solution, we assume there is an actor of type some_actor whose fields are persisted and used in the examples. See Non-Conditional inheritance actor fields.

109.2.1 Non-Conditional inheritance actor fields

These fields retain their values during Smart Replay. To ensure these values are persisted, follow the steps outlined below.

Let’s assume we want to persist the data for a field from the Object List (OL) custom data that represents a foo of type distance in OSC.

In the examples, all the files are stored in a directory in the environment variable $FTX_INGEST_SR_CONFIG.

109.2.1.1 Step 1: Add the field if it does not exist

If the data you want to persist in Smart Replay doesn’t already exist in the actor it belongs to, add it to a new separate file that you’ll be able to import in the ingestion OSC file as well as the Smart Replay configuration file. The field must be defined in both the ingestion and the Smart Replay scenario.

extend some_actor:
    foo: distance  # Only if the field doesn't already exist in the actor

109.2.1.2 Step 2: Mark the field to be persistent in Smart Replay

Set the fields as Smart Replay persistent using the smart_replay_persistent property (or an equivalent annotation when available).

This must be done in the same file where the new field is defined. Since this file will be imported into the Smart Replay configuration file, the field must continue to persist if a Smart Replay scenario is created from another Smart Replay scenario.

import "<file_with_enum_definition>.osc"

extend some_actor:
    foo: some_enum  # Only if the field does not exist in the actor
    properties(foo, smart_replay_persistent: true)

Example

extend vehicle:
    foo : string
    def post_sim_update() is also:
        if prev_state == null and state != null:
            for property in state.additional_properties:
                logger.log_info("Vehicle property: prop_key=$(property.name), prop_value=$(property.value)")
                var foo_property : string = sim_state.get_property_value_by_name("foo")
                if foo_property != "":
                    foo = foo_property

109.2.1.3 Step 3: Set the fields on the actor’s first frame

The fields that need to be persisted during Smart Replay should be set during the first frame of each reuse, as actors are reused.

Read the custom data from the OL and set the appropriate OSC value for each field that must persist in Smart Replay, ensuring the correct value is applied to the actor. Complete this before the end of the step in which the actor will appear.

Example

extend npc_vehicle:
    ingest_npc_vehicle_category: vehicle_category
    properties(ingest_npc_vehicle_category, smart_replay_persistent: true)
    properties(color, smart_replay_persistent: true)

extend plain_object:
    ingest_kind: object_kind
    properties(ingest_kind, smart_replay_persistent: true)

extend stationary_vehicle:
    ingest_stationary_vehicle_category: vehicle_category
    properties(ingest_stationary_vehicle_category, smart_replay_persistent: true)
    properties(color, smart_replay_persistent: true)

109.2.2 Conditional inheritance actor fields

In conditional inheritance, the issue described in the Technical limitation section arises. To preserve values for these fields in Smart Replay, an intermediate field is required to store the data.

For conditional inheritance, we assume that the actor has a generative field used for conditional inheritance.

109.2.2.1 Step 1: Create persisted replicas for actor conditional inheritance fields

Each conditional inheritance field requires a replica field to replace the original field used for conditional inheritance. This replica must be added in the same file to ensure the field is available in the Smart Replay run.

import "<file_with_enum_definition>.osc"

extend some_actor:
    foo: some_enum  # Only if the field does not exist in the actor
    properties(foo, smart_replay_persistent: true)

    ingested_bar: some_enum
    properties(ingested_bar, smart_replay_persistent: true)

import "<file_with_enum_definition>.osc"
extend some_actor:
    foo: some_enum  # Only if the field doesn't already exist in the actor
    properties(foo, smart_replay_persistent: true)
    ingested_bar: some_enum
    properties(ingested_bar, smart_replay_persistent: true) 
    properties(bar, smart_replay_persistent: false)

Example

import "sr_ingest_actors_data.osc"

extend npc_vehicle:
    def update_state(sim_state: av_actor_state) is also:
        if sim_state != null:
            for property in sim_state.additional_properties:
                logger.log_info("Vehicle property: prop_key=$(property.name), prop_value=$(property.value)")
                if property.name == "object_type":
                    if "OT_BASIC_VEHICLE" == property.value or "OT_POLICE_CAR" == property.value :
                        ingest_npc_vehicle_category = sedan

extend plain_object:
    def update_state(sim_state: av_actor_state) is also:
        if sim_state != null:
            for property in sim_state.additional_properties:
                logger.log_info("Vehicle property: prop_key=$(property.name), prop_value=$(property.value)")
                if property.name == "object_type":
                    if "OT_CHILD" == property.value:
                        ingest_kind = person

extend stationary_vehicle:
    def update_state(sim_state: av_actor_state) is also:
        if sim_state != null:
            for property in sim_state.additional_properties:
                logger.log_info("Vehicle property: prop_key=$(property.name), prop_value=$(property.value)")
                if property.name == "object_type":
                    if "OT_BASIC_VEHICLE" == property.value or "OT_POLICE_CAR" == property.value:
                        ingest_kind = stationary_vehicle
                        ingest_stationary_vehicle_category = sedan

109.2.2.2 Step 2: Set the replica fields on the actor’s first frame (per reuse)

This step is similar to Step 3: Set the fields on the actor’s first frame for non-conditional inheritance generative fields. The only difference in this step is that the value is set for the replica field, not the original field.

Example

extend npc_vehicle:
    def update_state(sim_state: av_actor_state) is also:
        if sim_state != null:
            for property in sim_state.additional_properties:
                logger.log_info("Vehicle property: prop_key=$(property.name), prop_value=$(property.value)")
                if property.name == "object_type":
                    if "OT_BASIC_VEHICLE" == property.value or "OT_POLICE_CAR" == property.value :
                        ingest_npc_vehicle_category = sedan
                    elif "OT_SEMI_TRAILER" == property.value or "OT_FIRE_TRUCK" == property.value:
                        ingest_npc_vehicle_category = semi_trailer_truck

109.2.2.3 Step 3: Add a constraint to your Smart Replay configurations to persist the original field

This step ensures that in Smart Replay scenarios, the original field used for conditional inheritance is set to the correct value from the ingested run.

You must add a constraint to ensure that the conditional inheritance field matches the replica field. This should be done in a separate file, as the constraint must apply only to Smart Replay scenarios, not the ingestion process.

import "$FTX_INGEST_SR_CONFIG/some_actor_ingest_data.osc"
extend some_actor:
    keep(bar == ingested_bar)

Example

import "sr_ingest_actors_data.osc"
import "$FTX/smart_replay/customers/abcd/sr_abcd_config.osc"

extend npc_vehicle:
    keep(ingest_npc_vehicle_category == vehicle_category)

extend plain_object:
    keep(ingest_kind == kind)

extend stationary_vehicle:
    keep(ingest_stationary_vehicle_category == vehicle_category)

109.2.2.4 Step 4: Import the constraint into all Smart Replay configuration files

You must import the constraint file into all Smart Replay configuration files so that it takes effect.

import "$FTX_INGEST_SR_CONFIG/some_actor_ingest_adapter.osc"

109.2.3 Alternative Solution

An alternative solution is to enable the creation of actors on the fly during a Foretify run. This ensures that actors are created with the correct conditional inheritance field values, preventing the issue from occurring.

However, this does not automatically resolve everything. You'll still need to set the fields from the OL custom data, as outlined in the steps under Non-Conditional inheritance actor fields.

109.3 Passing data from the real world to Smart Replay

An object in the real world can have many fields and attributes, which customers may want to pass through the Foretify ingestion run and use in Smart Replay to create a more realistic replay of the data.

Since the data can vary widely in terms of the field types, granularity, and relevance to the customer, we propose the following generic solution:

• Store the relevant attributes in the custom_data message within the object message in the object_list proto:

  proto 
  message Pair {
   string key = 1; 
   string value = 2; 
  }
  repeated Pair custom_data = 21;
  

• This message can be denoised to ensure consistency throughout the lifetime of the object. The data is then passed to the Foretify object, cycle by cycle, into the custom_data field in actor_data:

  struct actor_data:
     var id: uint
     var actor_label: string
     var exist_in_simulator: bool
     var custom_data: list of property
     var recording_id: list of recording_id_info
  

• The Application Engineer or Customer must write a translation layer to interpret the custom data and enforce the correct field values being set.

• Since actors can be reused and some fields cannot change during the run, we recommend creating twin fields for each relevant field the customer wants, using the sr_* prefix. This change will not affect the ingestion run, but the Smart Replay export will be able to read these fields and set the relevant values during the replay.