Skip to content

Using the Foretify Command-Line Interface

Foretify run execution flow

In order to understand the concrete situations that take place during test execution and also to debug eventual failures or gray-area behavior, it is useful to know the stages of the test execution and how different components interact with each other.

The first stage in the test flow is to load the OSC2 sources. These are loaded one by one, according to import order in the OSC2 Reference. The next step is to load and interpret the map that the scenario will use.

At this point Foretify is ready to plan the list of actions. This phase expands the behavior described in the scenario into finer granularity movement scenarios. It also chains various phases in the scenario.

If the planning phase succeeds, the test moves to simulation. In this phase the Foretify engine starts interacting with the simulator:

  • Advancing the simulation
  • Retrieving the actors’ dynamic information (position, speed, acceleration)
  • Adjusting the plan, if necessary, in order to achieve the plan’s goals.

You must keep in mind that in any AV simulation there is a grain of unpredictability: SUT can take unpredictable decisions to handle situations when forced by the traffic context. This means that the plan does not always match reality. Foretify alleviates the side effects of unexpected SUT behavior by adjusting the plan at the end of each simulation step.

Executing a single run interactively

To execute a single interactive run with the Foretify GUI:

  1. Load the test file into the Foretify GUI:

    Shell command: invoke Foretify with GUI
    $ foretify --load <test-file> --gui

  2. In the Terminal Pane, type the run command.

  3. To exit Foretify, enter the quit command.

To execute a single interactive run using the Foretify Command-Line Interface:

  1. Load the test file into Foretify:

    Shell command: invoke Foretify with CLI
    $ foretify --load <test-file>

  2. At the foretify> prompt, type the run command.

Executing a single run in batch

To execute a single batch run, invoke Foretify with the --load, --run and --batch options.

Shell command: invoke Foretify in batch
$ foretify --load <test-file> --run --batch

Viewing run data

For each run, Foretify creates a new folder under the <work_dir>/runs folder. The timestamp is used as the name of the folder.

The folder contains all the information required in order to recover a previous run, either for debugging purposes or in order to re-execute the run.

Resources included in the <work_dir>/runs/<timestamp>/ folder are:

  • source/ is a directory containing all the OSC2 files loaded into Foretify during the run, including user source files.
  • xviz_data/ is a directory containing zip files with the Visualizer data from the run.
  • debug_data.pb.gz and run_data.pb.gz are internal Foretellix files that are used by Foretify and Foretify Manager for coverage calculation and single-run debugging.
  • public_data.pb.gz contains the raw data collected during the Foretify run.
  • run_info.json contains the metadata for the run.
  • run.log is the log file for the run.

You can use the public_data.pb.gz file to quickly access the information in its raw format before it is processed by Foretify Manager. There are two methods to access the information in the public_data.pb.gz files:

  • Use a programming language to access the data.

    The Foretify release includes a file $FTX/interfaces/run_results/public.proto that defines the schema of the run results.

    Using the public.proto file, you can generate the required header files using the Protobuf compiler, For information on how to do this, see Protocol Buffers Tutorials.

  • Use Foretify's run_results_parser to convert the binary results into a text-based JSON file.

    Shell command: convert the public data file
    $ $FTX/bin/run_results_parser/run_results_parser <path to run folder>

    After executing the script, the run folder contains a new file called public_data.json. You can be read the file with any text editor application.

Re-executing a run

You can recover a run by selecting it in the History tab of the Foretify GUI, or by loading it during invocation using the --load_run option:

Shell command: re-execute a run
$ foretify --load_run <run_folder>
Foretify can load a test run even if you have moved the folder to a different path or a different server.

Note

When re-executing a run, Foretify does not read from the run's source directory, but instead from the Protobuf file (<rundir>/debug_data.pb.gz). This means that changing the files in the source directory for the purposes of a rerun is not supported. See Viewing run data for details about run directories.

When you load a run that was executed with a Foretify version different than the current one, you receive a warning specifying that re-executing the run might yield different results.

Recording a run for later replay at Foretellix

You can record a Foretify test execution in your environment, and send the run results to Foretellix for reproduction. This shipped data enables Foretify to reproduce the entire run on the Foretify side without depending on other components, such as the SSP/Simulator and DSP/SUT.

To record a test run, set the option debug_record at Foretify startup.

Shell command: invoke Foretify with record on
$ foretify [other options] --debug_record
YAML code: set record on
runtime:
  debug_record: true

All test run folders created when Foretify is executed in this mode contain an additional recording file that allows reproduction of the run by Foretellix. Please ship the generated run folder (created under <work_dir>/runs/) to Foretellix.

At Foretellix, invoke Foretify wtih the debug_replay option to re-run the recorded test. You can load additional OSC2 files to help with the debugging if you want.

Shell command: replay a recorded run
$ foretify --debug_replay <path to received folder>

You can view the re-run results in log, Visualizer, and any other debug method such as trace.

Recording simulator videos

If you use the CarMaker or Carla simulator, you can record a video of your Foretify test execution. To record a video of a Foretify run with CarMaker or Carla, configure your test to use the simulator, then use the --record_video option on the command line:

Shell command: invoke Foretify and record a video
foretify --load <osc_file> --record_video

If the Foretify run folder is accessible to the Simulator Support Package (SSP), the mp4 is written to the run folder. For CarMaker, if this folder can't be accessed by the SSP, for example, if you're running on a separate machine, then the mp4 is written to the SimOutput folder of the CarMaker project. The output filename has the Foretify run ID as its prefix.

See also the record_video documentation.

CarMaker recording prerequisites and limitations

  • CarMaker must be running in GUI mode or CarMaker for Simulink (CM4SL).

  • The CarMaker TCP ScriptControl port must be opened and set to port 64791. For regular CarMaker, set this in the .CarMaker.tcl startup script. For CM4SL, set this with the MATLAB command CMData.CmdPort = 64791;.

  • To export video, IPGMovie must be running and connected at the end of the test. IPGMovie should be automatically started by the .CarMaker.tcl startup script.

  • If the SSP is run remotely and the video is not written to the run folder, Foretify will not be able transcode the video to h264.

See the CarMaker Programmer's Guide for information about using IPGMovie.