78. FRun reference

78.1 Template syntax

A template file (default extension .tosc) is a text file with a format similar to that of an OSC2 file. It consists of a template header, followed by OSC2 code with embedded parameters. Here is the full format:

$$template():
    <params-block>
$$expand():
    <OSC2-code-block>

78.1.1 The <params-block>

This block is indented four spaces from the previous line. It consists of

• Parameter definitions
• Connect-to-cover definitions
• Blank lines
• Comments beginning with the # character

78.1.1.1 Parameter definitions

The parameters you define should include all explicitly constrained attributes of the scenarios executed in the test.

<param-name>: $$param(["<default>"])

<param-name> is used in the parameter-names line of the run table.

<default> is the default value of this parameter. A parameter without a default value is called an optional parameter, meaning the test can be run without any constraints on this parameter.

sim: $$param("sumo") # The simulator name to be used for this test (sumo or carla)
map: $$param("hood") # The map to be used for this test (hood, hooder etc.)

78.1.1.2 Connect-to-cover definitions

To allow the fill option for a parameter, you must define a connect-to-cover for that parameter. This construct associates a parameter with a cover item. The cover item defines the legal values or ranges (buckets) for the parameter.

$$connect_to_cover(<name>, <cover-item>)

$$connect_cover(map, top.main.end.map) # a fake cover item
$$connect_cover(side, sut.cut_in_and_slow.end.side) # an actual cover item

Example 1: no fill template

This example is a simple template for the cut_in_and_slow() scenario, where the template user can change the simulator, the map and optionally the side parameter.

# cut_in_and_slow.tosc
$$template():
    sim: $$param("sumo") # The simulator name to be used (sumo or carla)
    map: $$param("hood") # The map to be (hood, hooder etc.)
    side: $$param() # Whether to go left or right
$$expand():

import $FTX/env/basic/exe_platforms/$$(sim)_ssp/config/$$(sim)_config.osc
import $FTX_PACKAGES/base_scenarios/scenarios/vehicle_cut_in/vehicle_cut_in_and_slow/vehicle_cut_in_and_slow_top.osc

extend test_config:
    set map = "$FTX_PACKAGES/maps/$$(map).xodr"

extend top.main:

    do a: cut_in_and_slow()
        keep(a.side == $$(side))

Example 2: using fill template

This example shows how to enable fill for both the side parameter and the map parameter. Note that the scenario does not define coverage for map, so the template must provide a fake coverage definition for map. No coverage is collected for map.

# cin_slow_fill.tosc
$$template():
    sim: $$param("sumo") # The simulator name to be used (sumo or carla)
    map: $$param("hood") # The map to be (hood, hooder etc.)
    side: $$param() # Whether to go left or right
    $connect_cover(map, top.main.end.map)
    $connect_cover(side, sut.cut_in_and_slow.end.side)

$$expand():

import $FTX/env/basic/exe_platforms/$$(sim)_ssp/config/$$(sim)_config.osc
import $FTX_PACKAGES/base_scenarios/scenarios/vehicle_cut_in/vehicle_cut_in_and_slow/vehicle_cut_in_and_slow_top.osc

type map_name: [highway, long_single_road, long_single_road_2, cloverleaf, straight_long_road, hooder]

extend test_config:
    set map = "$FTX_PACKAGES/maps/$$(map).xodr"

extend top.main:
    # Add a map-name field and cover it, just so we can fill it
    map_name: map_name with: keep(it == $$(map)); cover(it)

    do a: cut_in_and_slow()
        keep(a.side == $$(side))

78.1.2 The <OSC2-code-block>

This block is OSC2 code with embedded parameters in the format $$(<name>). The embedded parameters are handled as follows:

• Any $$(<name>) is replaced by the value of the parameter called <name>.

• If a parameter does not have a value, any line where $$(<name>) appears is removed.

• To remove a line of code that is associated with a parameter but does not have (\((<name>)** in it, add **\)\)(<name>) in a comment on that line.

keep(c.side == $$(side))

in c.turn_left.car1 with: # For $$(lane)
    lane($$(lane))

In the example above, both lines are removed if $$(lane) has no value.

78.1.2.1 Constraint handling

In the OSC2 code block, use the following syntax to specify a constraint.

keep(<param-pathname> == $$(<param>))

The template plugin handles the conversion of the constraint to the proper syntax, if required because of the user-supplied value.

keep(c.speed == $$(speed))

keep(c.speed in [3..5]kph)

keep(soft c.speed in [3..5]kph)

78.2 Template syntax for sub templates

78.2.1 Create a sub template or post template

The syntax for sub templates and post templates is the same as the main template syntax, except that these templates begin with (\(sub(<role-type>)** instead of **\)\)template().

The <role-type> parameter must match the <role-type> parameter defined for the sub template in the <sub-template-params-block> of the main template.

$$sub(role-type):
    <params-block>
$$expand():
    <OSC2-code-block>

# manually_driven_kinematic_no_ADAS_config.tosc
$$sub(config):
    sim: $$param()
    map: $$param()

$$expand():

import "$FTX/config/sim/$$(sim)_default.osc"

extend test_config:
    set map="$FTX_PACKAGES/maps/$$(map).xodr"

78.2.2 Use a sub or post template in a main template

A main template file that uses sub or post templates consists of a template header, followed by OSC2 code with embedded sub templates. Here is the full format:

$$template():
    <sub-template-params-block>
$$expand():
    <sub-template-code-block>

78.2.2.1 The <sub-template-params-block>

This block is indented four spaces from the previous line. It consists of a list of one or more sub or post templates in the following format:

$$template():
    <role-name>: $$sub_table(role: <role-type>)

• <role-name> must match the <role-name> parameter passed by $$sub() in the expansion section of this main template.

• <role-type> must match the <role-type> parameter passed by $$sub() in the sub or post template. It must also match a column header in the main run table. FRun checks that an appropriate sub table is listed in the specified column of the main table. If you place a sub table in the column for a different role type, you see an error.

Given this information, FRun can determine the sub template that needs to be expanded by finding the appropriate run group in the sub table.

# lead_vehicle.tosc
$$template():
    config: $$sub_table(role: config)
    lead_vehicle: $$sub_table(role: lead_vehicle)

78.2.2.2 The <sub-template-code-block>

In the $$expand() section, you must expand the sub template using the following syntax:

$$expand():

# the sub templates should be expanded in the appropriate order and indentation
$$(<role-name1>)
$$(<role-name2>)

Although each sub template is expanded in place, any import statements in it are moved to the top of the test file.

78.3 Do not expand both test parameters and sub templates in a single template

A template can expand either sub templates or regular parameters. Creating a template that expands both is not allowed. For example, the following combination of expansions in a template is not allowed:

// expanding regular parameters and sub-templates together is not allowed.
$$expand():
    $$(my_config)      // a sub-template
    ...
    keep(cutin_car.color == $$(color)) // a regular parameter

78.4 Run table format

FRun run tables are defined in an Excel or CSV file. The file can contain only the following:

• Comments: Place two forward slashes // at the beginning of the comment text. Cells in a row where the first column starts with // are ignored. Also, if // appears in a column, then all text in that column is ignored.

• Empty rows and columns: Empty rows and columns are included in the line and column counts.

• Run tables: You can define one or more run tables (run groups) in a single file. These run groups define the test suite. Run tables are separated by at least one empty row.

78.4.1 General run table syntax

Each run table consists of a two-line header, followed by zero or more parameter lines. If there are zero parameter lines, then there are no runs in the group.

The plugin line (the first header line) has the following format:

• Column 1: *<run-group-name>

  Run group names should be proper names (letters, digits and underscores). A run group name must be unique within the CSV file. The asterisk * is required.

• Column 2: <plugin-name>[ <main-param>]

  <plugin-name> is the file name of the plugin. The default extension is .py, and the file is searched along $OSC_PATH. For plugins that have a main parameter, it appears after the plugin name. For example, the basic plugin has no main parameter, but the template plugin has one – the name of the template file.

  Note

  Different run tables in the same CSV file can have different plugins.

• Columns 3 and onwards: <param>: <value>

  Each column starting with Column 3 can contain a single value for a single parameter. Parameters defined in this manner are called common parameters and are shared across all tests created by this run table.

The parameter-names line (the second header line) has the following format:

• Each column contains the name of a parameter. The template and basic plugins have at least two parameters:

  • The runs parameter specifies how many times to run each test created by this line. (Default: 0)

  • The seed parameter specifies the seed to be used for the first test. Legal values are any number or the word random. If the runs parameter is greater than 1, then for each subsequent run the seed is incremented. (Default: 1)

• No empty columns are allowed until after the last parameter is specified.

• The parameter name can only contain letters, numbers, underscores and the period character (.)

  Note

  The template plugin allows the period character only in the special keep and refine parameter names.

• The parameter name determines the meaning of the column in the parameter lines below.

The test specification lines (zero or more) have the following format:

• Each column in the line is empty or contains a value for the parameter named in the parameter-names line.

78.4.2 Run table syntax for the template plugin

The format of a template run table is as described in General run table syntax. To activate the template plugin, enter the following in Column 2 of the plugin line:

template <template-filename>

If this is the run table for an all-in-one template, you must assign values to the runs and seed parameter. In addition, you can assign values to any parameters defined in the template. For an example, see Example: all-in-one template.

If this is the run table for a main template that uses sub or post templates, you must assign values to the runs and seed parameter. In addition, for each sub or post template, you must specify the role type of that template as a column header. (The role type of the sub or post template is defined in the main template.) Then, specify in that column the name of the run group defined in a run table for that sub or post template. Example: sub templates.

78.4.3 Run table syntax for the sub plugin

The format of a sub run table is as described in General run table syntax. To activate the sub plugin, enter the following in Column 2 of the plugin line:

sub <sub-template-filename>

78.4.4 Run table syntax for the post plugin

The format of a post run table is as described in General run table syntax. To activate the post plugin, enter the following in Column 2 of the plugin line:

post <post-template-filename>

78.4.5 Run table syntax for the basic plugin

The basic plugin is used to run specific tests a specific number of times. It has two parameters in addition to runs and seed:

• test is the test file name
• directory is the pathname of the directory containing the test file

78.4.6 Methodology tips for template developers

78.4.6.1 Using comments

Template users might not have access to the template code or to the code for scenarios invoked. It is recommended to use comments liberally to explain the following:

• Any default values the template sets for parameters.

• The correct path to be used for special parameters.

• Any additional information necessary to understand how to debug test results, such as scenario phase names.

78.4.6.2 Using defaults

Defining a default value for parameters used in most tests is recommended. For example, most tests can use the same simulator and perhaps the same map. If these defaults are explained in a comment, template users can easily see whether they need to change the defaults for a particular set of tests.

78.4.6.3 Using special parameters

Defining at least one special keep parameter and one special refine parameter is a good way to illustrate the correct path required for similar parameters.

78.4.6.4 Using soft constraints

Using hard constraints rather than soft constraints in the template is recommended. The template users have the option of specifying a soft constraint or otherwise modifying the constraint if a contradiction occurs.

78.4.6.5 Using the pass statement

When a statement uses an FRun parameter and that parameter is not set, then the statement is not generated and is left empty. The pass statement lets you avoid an error when all the statements in that block are not generated (so the block is empty).

78.5 FRun invocation

Purpose

Invoke one or more runs interactively or in batch, specifying parameter values via a run table

Syntax

frun [-h] [--help]
     [--work_dir output directory]
     [--run_name name]
     [--debug] [--info]
     [--split_executions]
     [--line [filename:]line_number] [--group group_name]
     [--int ["foretify_args"]]
     [--server host:port]
     [--fmanager_host host] [--fmanager_port port]
     (--csv csv_file | --table file | --excel xls_file |
      --ods ods_file | --file txt_file )
     [--create] [--batch] [--filter] [--skip_filter]
     [--runner plugin_name] [--runner_path path]
     [--runner_args ["options"]]
     [--foretify_config_file [config.yml]]

-h, --help

Print the list of invocation options.

--work_dir <work-dir-name>

Define the name of the session directory. The default is ~/foretify/sessions/<timestamp>_<frun-pid>.

--run_name "name"

Define a string to be included in the FRun work_dir name. For example, if the name is specified as "BSM_reg341", the work directory name is <timestamp>_<frun-pid>_BSM_reg341. The name is intended to help you quickly identify a test suite within a work directory.

--debug

Write out information to aid in debugging FRun execution.

--info

Print a summary of FRun execution at the end of the test suite.

--split_executions

If a run crashes in a way that doesn't let Foretify finish the test suite, you can use this option to ensure that the test suite doesn't hang.

--line [<filename>:]<line-num>

Include in the test suite just the plugin activations on this parameter line. Line numbers are the row numbers within the spreadsheet. Blank lines are included in the count. If you are passing more than one file, the name of the file must be specified wit the line number, for example:

frun --csv a.csv --csv b.csv --line 3 # error: multiple files
frun --csv a.csv --csv b.csv --line a.csv:3 # valid (selects line 3 from  a.csv

--group <group-name>

Include in the test suite just the plugin activations in the specified group.

--int [=“<foretify-args>”]

Run a test interactively in Foretify. <foretify-args> are any options allowed in Foretify activation, such as --gui. This option requires that you also specify a parameter line in the run table with --line.

--server <host>:<port>

Specify the host and port for the Foretify server.

--fmanager_host "<host>" --fmanager_port "<port>"

Specify the host and port for the Foretify Manager server. This option causes the host name and port to be included in the collect_runs.sh script.

--csv <csv-file-name>

Specify the file name of a run table file in CSV format. You can specify this option multiple times with different files.

--table <file-name>

Specify the file name of a run table in Excel, OpenDocument or CSV format. FRun determines the format depending on the file extension. You can specify this option multiple times with different files.

--excel <xls-file-name>

Specify the file name of a run table in Excel format. You can specify this option multiple times with different files.

--ods <ods-file-name>

Specify the file name of a run table in OpenDocument format. You can specify this option multiple times with different files.

--file <txt-file-name>

Load a text file containing multiple --excel, --ods or --csv options. This option is useful when you are defining a test suite based on multiple files. You must specify additional options like this:

frun --file my_file.txt --batch

--create

Create the test files and the run_requests.json file without sending the requests to the dispatcher. This option is useful if you want to inspect the test files or if your CI infrastructure invokes FRun and then does the right thing with the requests.

--batch

Run this set of tests in batch mode using the FRun dispatcher.

--filter

Invoke Foretify to do the initial test generation and planning, and then exit without executing the run. This option is useful if you want to inspect tests that failed initial generation and planning.

--skip_filter

Use this option with --batch if you are re-executing a test suite and are confident that all tests will compile and generate without error.

--runner plugin_name

Specify the name of a runner.

--runner_path path

Specify the path of a runner.

--runner_args "<options>"

Specify arguments to be passed to the runner.

--foretify_config_file <config.yml>

Specify a YAML configuration file to be passed to Foretify. For example:

runtime:
    set:
        - "config.sim.enable_gui=false"

For more information on Foretify configuration files, see the Foretify User Guide.

Description

FRun is a shell script with two main use modes, interactive and batch. There are additional modes that just create the test files or create and filter the tests.

FRun invocation examples

Interactive mode: Execute one or more interactive foretify runs according to a specific line in the table. For example:

$ frun --csv my_table.csv --line 15 --int=“--gui --run”

$ frun --csv my_table.csv --batch --skip_filter

Create mode: Create the session directory, the test files and a run_requests.json file for inspection or for running via some other CI infrastructure. For example:

$ frun --csv my_table.csv --create