Skip to content

79. Test Suite Manager Python Library

The TSM Library enables writing python "TSM-lite" implementations, focusing on the TSM-specific logic, abstracting and simplifying common requests without having to invest in some integration details.

A typical use of the TSM Library is to write various optimizer and analytic TSMs. Such TSMs (and others) have a common workflow that can be generalized in the following graph.

While the general workflow is the same, the data required by a TSM to run will vary depending on its specific implementation and purpose.

This section details the Test Suite Manager (TSM) API’s methods, usage examples and recommended practices to utilize the Test Suite Manager Python Library (also termed the TSM Library).

Note

This section provides information about the TSM Library only. It does not outline usage of the lower level APIs it makes use of (i.e., Foretify Manager APIs).

79.1 Key terms for the Test Suite Manager

Following are key concepts introduced with the Test Suite Manager.

Test Suite
A group of tests run together as a single job in Foretify Manager, previously termed regression.
Test Suite execution (TSE)
An execution of a test suite.
Test Suite Manager (TSM)
A high-level component that enables easily defining and running of test suites in various ways. Any specific implementation of a TSM implements a specific behavior out of a wide range of possibilities. There is no definite list of options, but optimizers are a notable usage example.
Foretify Manager Python API
A python library that exposes various functions of Foretify Manager. The library makes use of this API behind the scenes, but it is also available for users and can be easily accessed for any functionality the library doesn’t provide. See Foretify Manager Python API.

79.2 Import the TSM Python Library

To use the TSM library on a dev machine, it must be installed from a wheel file.

python3 -m pip install /<some_path>/tsm_api-<version_str>.whl

To use the library in python code, a single import is required:

from <tms_lib_path> import *

79.3 TSM Python Library API Overview

The library interface is built from two main components, TSMClient and message classes.

79.3.1 TSMClient

TSMClient is a python class that exposes the various endpoints of the API for all the required interactions with Foretify Manager (also referred to as FManager) for running Test Suites and getting results with minimum effort.

With the exception of the connect, version and init methods, a method that does operation X is named x, takes a single parameter of the type XRequest and returns a single parameter named XResponse. They all have the following signature:

@staticmethod
def x(x_request: XRequest) -> XResponse:

79.3.2 Message classes

All of the message classes are not created using their constructors. Instead, instances are created via classmethods. The library uses this convention to essentially have multiple “constructors” per class. The message classes can be divided into two types, request-response message pairs and data/utility messages.

79.3.2.1 Request-Response Message Pairs

These messages come in pairs corresponding to most of the TSMClient’s methods.

Request Messages

The request messages contain all of the information required to perform the requested action.

Requests for running test suites or getting data from workspaces always require the TSM ID of the relevant workspace.

Response Messages

All response messages contain a status, which can be obtained with the status property. The status contains an is_ok property that is True if the request completed successfully, in which case the response can be queried for the relevant information. If the is_ok property returns a False value, the status can be queried for further information about the failure.

The response messages themselves have an is_ok property that can be used for convenience (i.e., x_response.is_ok instead of x_response.status.is_ok).

79.3.2.2 Data/Utility Messages

These messages are meant to hold data and/or provide information to the client. They are used as building blocks for the Request-Response messages.

Do not use message constructors

All message classes contain class methods to easily create them or to query their contents (details below in each message’s documentation).

You should only use these methods for constructing instances. Do not call the message classes constructors directly.

79.3.3 Concepts

79.3.3.1 Workspaces and projects

All test suites run via the API are automatically added to a workspace upon successful execution. To run Test Suite Executions (TSEs) via the API, you must set up a workspace (or multiple workspaces) that you are going to use for execution.

The API provides two different options for this purpose, loading an existing workspace to add new TSEs, or requesting to create a new workspace to run TSEs in.

Workspaces exist in Projects in Foretify Manager, and to create and modify workspaces in a project, the user whose credentials are used via the API must have access to the relevant projects.

Project creation is not possible via the API; they have to be created beforehand with an access level of OWNER or EDITOR for the user credentials that will be used with the API. It can be specified either by ID or by name.

Once a workspace has been loaded or a new one is requested, an internal TSM ID is assigned by the API for that workspace and it is returned to the user. Configurations for running FRun-based test suites (using existing FRun tables) or template-based test suites (to programmatically set template parameter values) can be loaded for this TSM ID. The loaded configurations also get IDs that are used to request running test suites.

It is possible to create and/or load multiple workspaces in multiple projects in a single TSM run to suit your needs with every TSE automatically added to the workplace specified in the request upon successful execution.

79.3.3.2 OSC Templates and Sub-Templates configurations

OSC Templates and Sub-Templates are a powerful tool for running the same scenario with different input parameters. This is extremely useful for various objective optimizers which require control of the inputs of the scenario.

However, writing an optimizer that knows how to work with all template scenarios is hard, since they vary widely in their parameters and their KPIs. Parameters may be of different types, have required resolutions for their possible values, different valid ranges and values for them, and so on.

To make generalization easy, the TSM Library introduces a general configuration mechanism for templates and sub-templates. This configuration mechanism abstracts away the specifics of the templates and sub-templates being used. This allows writing TSMs that utilize generic algorithms.

Template and Sub-Template configurations are stored as files in a json format. Using files allows us to easily reuse such configurations. See the OSC Template Configuration Files code samples as a reference to their format.

Also see the OscTemplateConfig and OscSubTemplateConfig documentation for further information on the data they contain.

79.3.3.3 FRun CSV

FRun CSV is a message containing input for an FRun job’s input. This is either a list of CSV files, or a single file’s path that contains such a list (See details in the FRunCSV message documentation).

79.3.3.4 Test groups

A test group is a group of test runs that have the same input parameters except for the seed which may have a different value.

The API exposes methods to get test groups from all test runs in a workspace or from specific TSEs in a workspace. It also exposes an option to rerun test groups any number of times with newly provided or random seeds.

79.3.3.5 Coverage

The TSM API provides methods to get coverage info for specified coverage items from a workspace.

79.4 Code samples

The following code samples show how to use the library. Each code sample is a continuation of the previous one.

79.4.1 Using the API

79.4.1.1 Initialize a TSMClient

Before doing anything with the library, you need to initialize a TSMClient and connect it to Foretify Manager.

# Your own imports
from python_api import tsm_client
from python_api.tsm_messages import *


def main():
    client = tsm_client.TSMClient(
     local_work_directory='/work/directory/optimizers/workdir')


    [success, error] = client.connect(
        '8.0.8.1', 8080, get_user())
    if not success:
        # TODO: Error handling.
        print(error)
        return


   # client is connected to Foretify Manager and can now be used.

79.4.1.2 Specify a workspace

Creating a workspace or loading an existing one is required, since that’s how we associate all TSEs run via the API to a workspace. Multiple workspaces can be created/loaded to run various TSEs.

79.4.1.3 Create a new workspace

The following code sample creates a new workspace with the given name in Foretify Manager as soon as the first Test Suite is run for the workspace’s TSM ID.

create_workspace_response = client.create_workspace(
   CreateWorkspaceRequest.create(workspace_name='My new workspace',
                                 project_id_or_name=1))


if not create_workspace_response.is_ok:
    print(create_workspace_response.status.error_message)
    return
# Now we get the TSM ID so we can use it to load some OSC Template Configurations and/or FRun CSV Files.
tsm_id = create_workspace_response.workspace_info.tsm_id

79.4.1.4 Load an existing workspace

We can also change the first method call to use an existing workspace from Foretify Manager instead, and continue similarly.

load_workspace_response = client.load_workspace(
   LoadWorkspaceRequest.create(workspace_id='abcdef0123456789'))

79.4.1.5 Load an OSC Template Config

We now load an OSC Template Configuration for the workspace we created/loaded. This configuration will be stored by the TSM and enable the TSM Client to run Template Test Suites using it.

The return value of this call allows the client’s code to query the relevant parameters and based on their type and allowed range/values, create test inputs for them dynamically.

load_config_response = client.load_osc_template_config(
    LoadOscTemplateConfigRequest.create(
        tsm_id,
        '/work/directory/optimizers/template_config.json',
        'foobar_template_env'))


if not load_config_response.is_ok:
    # TODO: Error handling.
    print(load_config_response.status.error_messages)
    return

79.4.1.6 Create a RunTemplateTestSuiteRequest

Now that we’ve loaded an OSC Template Configuration, we can run as many Template Test Suites as we like using that configuration. To do that, we create a RunTemplateTestSuiteRequest.

We can use the return value from loading the Osc Template Configuration to do this.

run_test_suite_request =
    RunTemplateTestSuiteRequest.from_load_osc_template_config_response(
        load_config_response)

Alternatively, we can pass the TSM and Osc Template Configuration IDs explicitly.

run_test_suite_request = RunTemplateTestSuiteRequest.create(
    tsm_id=tsm_id, config_id=load_config_response.config_id)

79.4.1.7 Create and add test inputs to our RunTemplateTestSuiteRequest

Now that we have the request created, we can add test inputs to run as part of the TestSuite. To do this, we iterate over all the relevant parameters from the loaded configuration.

template_config = load_config_response.osc_template_config
tests_count = 10
for i in range(tests_count):
    test_input = TestInput.create(TestId.from_number(i))
    for [param_name, parameter_config] in template_config.get_parameters(
        OscTemplateConfig.PhysicalParamFilter.NO_FIXED,
            OscTemplateConfig.StringParamFilter.NO_DEFAULTS).items():
        # Add an input depending on parameter configuration:
        if parameter_config.has_string_parameter:
            # Choose the first value
            param_input = ParameterInput.from_string(
                parameter_config.string_parameter.values[0])
        else:  # This is a physical parameter
            physical_parameter = parameter_config.physical_parameter
            # Let's use the default if there is one.
            if physical_parameter.has_default_value:
                param_input = ParameterInput.from_numeric_value_or_range(
                        physical_parameter.default_value)
            else:
                # We don't have a default, so let's choose a
                # different fraction of the range for each of the inputs
                (min, max) = physical_parameter.valid_range.min_max
                fraction: float = (max - min) / tests_count
                param_input = ParameterInput.from_min_max(
                    min=min + fraction * i,
                    max=min + fraction * (i + 1),
                    rounder=physical_parameter.resolution_rounder)
        test_input.add_parameter(param_name, param_input)
    # Finished adding parameter inputs to the current test input.
    # Add the test input to the test suite request.
    run_test_suite_request.add_test_input(test_input)

79.4.1.8 Run a Template Test Suite

Now that we added all of our inputs to the Test Suite request, we can run it.

# We finished adding all test inputs to the request. Now we run it.
template_test_suite_result = client.run_template_test_suite(
    run_test_suite_request)


if not template_test_suite_result.is_ok:
    print(str(template_test_suite_result.status))
    return

79.4.1.9 Iterate over the test results for all test runs in the Test Suite

The return value contains the results for all the Test Inputs we added to the Test Suite. The Test IDs for the results match those of the Test Inputs.

In this example, we simply print them, but a typical use case would gather various outputs to calculate the new inputs for another TestSuite run or stop if a certain target function is met.

for test_output in template_test_suite_result.test_outputs:
    print(f'Test ID: {test_output.id.number}')
    print('  KPIs: ')
    for [name, value] in test_output.results.items():
        print(f'    {name}: {value}')
    print('  Parameters: ')
    for [name, param_output] in test_output.parameters.items():
        if param_output.has_string:
            print(f'    String parameter "{name}": {param_output.string}')
        else:
            numeric_output = param_output.numeric
            print(f'    Numeric parameter "{name}":')
            print(f'      Requested: {numeric_output.requested}')
            print(f'      Generated: {numeric_output.generated}')
            print(f'      Actual:    {numeric_output.actual}')

79.4.1.10 Load a FRun CSV File

To run FRun Test Suites, we need to load a FRun CSV File. This is the same as test suites run from the UI by choosing CSV files and an environment settings definition in Foretify Manager. 

load_frun_csv_response = client.load_frun_csv(LoadFRunCSVRequest.create(
    tsm_id,
    '/work/directory/frun_inputs/some_vehicle_maneuver.txt',
    environment_settings_name='foobar_frun_env'))


if not load_frun_csv_response.is_ok:
    # TODO: Error handling.
    print(load_frun_csv_response.status.error_message)
    return

79.4.1.11 Create a RunFRunTestSuiteRequest

Now that we’ve loaded a FRun CSV, we can run as many FRun Test Suites as we like with it. To do that, we create a RunFRunTestSuiteRequest.

We can use the return value from loading the Frun CSV to do this.

run_test_suite_request =
    RunFRunTestSuiteRequest.from_load_frun_csv_response(
        load_frun_csv_response=load_frun_csv_response)

Alternatively, we can pass the TSM and FRun CSV IDs explicitly.

run_test_suite_request = RunFRunTestSuiteRequest.create(
    tsm_id=tsm_id,
    frun_csv_id=load_frun_csv_response.frun_csv_id)

79.4.1.12 Run a FRun Test Suite

Now that we created the request, we can run it. The result only contains the status and the Foretify Manager Test Suite ID.

# run the frun test suite request
frun_test_suite_result = client.run_frun_test_suite(
    run_test_suite_request)


if not frun_test_suite_result.is_ok:
    print(str(frun_test_suite_result.status))
    return


test_suite_id = frun_test_suite_result.fmanager_test_suite_id
print(f'FManager Test-Suite ID: {test_suite_id}')

79.4.2 OSC Template Configuration Files

Check the OscTemplateConfig and OscSubTemplateConfig documentation for more information on the meaning of each of the values in this example.

79.4.2.1 Simple OSC Template - Single File Configuration

This configuration example is for an OSC Template that doesn’t use Sub-Templates. The entire configuration is a single file.

Note

Some configurations can be determined automatically when a workspace is provided. For more information, see LoadOscTemplateConfigRequest documentation.

{
  "templatePath": "$FTX/templates/my_template.tosc",
  "results": [
    "min_ttc",
    "max_sut_speed",
    "min_distance",
    "max_deceleration"
  ],
  "maxTestDurationSeconds": 300,
  "parameters": {
    "params": {
      "speed": {
        "physicalParameter": {
          "validRange": {
            "min": 20.0,
            "max": 120.0
          },
          "unit": "kph",
          "default": {
            "range": {
              "min": 50.0,
              "max": 60.0
            }
          },
          "generatedRecordName": "sut_gen_speed",
          "actualRecordName": "sut_actual_speed"
        }
      },
      "npc_color": {
        "stringParameter": {
          "values": [
            "red",
            "blue",
            "green"
          ],
          "default":{
            "value": "red"
          }
        }
      },
      "lead_duration": {
        "physicalParameter": {
          "validRange": {
            "min": 2.0,
            "max": 10.0
          },
          "unit": "s",
          "resolution": 0.02,
          "fixed": {
            "value": 4.0
          },
          "actualRecordName": "lead_time_actual"
        }
      },
      "slow_duration": {
        "physicalParameter": {
          "validRange": {
            "min": 2.0,
            "max": 10.0
          },
          "unit": "s",
          "resolution": 0.02,
          "generatedRecordName": "lead_time_gen"
        }
      }
    },
    "coverage_item_paths": {
      "dut.some_scenario.end.min_speed": {
        "cover_params": ["lead_time_gen"] },
      "dut.some_scenario.end.lead_and_slow_durations": {
        "cover_params": ["lead_duration", "slow_duration"] },
      "dut.some_scenario.end.min_ttc": {}
    }
  }
}

79.4.2.2 OSC Template with Sub Templates - Multiple Files Configuration

This configuration example is for an OSC Template that uses Sub-Templates where the configuration is split into multiple files.

79.4.2.2.1 Main OSC Template Configuration

This is the configuration for the topmost OSC Template, the one that combines Sub-Templates together to create a scenario.

Instead of containing a params map for the template’s parameters, it will contain a subTemplatePlaceholders that contains additional information about the subtemplates used and where their configuration files are located.

Note

Roles are optional. They act as an advanced feature allowing to specify information used by the inference mechanism

for missing information. The role is expected to be a struct (e.g., sut.cut_in), 'config' or be left empty. The behavior for each is described below under Template Config Inference.

{
  "templatePath": "$FTX/templates/my_template.tosc",
  "maxTestDurationSeconds": 300,
  "subTemplatePlaceholders": {
    "subs": {
      "sub1": {
        "role": "role1"
      },
      "sub2": {
        "role": "role2"
      }
    },
    "subTemplateConfigPaths": {
      "paths": {
        "sub1": "$TEST_ROOT/test/sub_template_config1.json",
        "sub2": "$TEST_ROOT/test/sub_template_config2.json"
      }
    }
  }
}

{
  "templatePath": "$FTX/templates/my_template.tosc",
  "maxTestDurationSeconds": 300,
  "subTemplatePlaceholders": {
    "subs": {
      "sub2": {
        "role": "role2"
      },
      "sub1": {
        "role": "role1"
      }
    },
    "subTemplateConfigPaths": {
      "paths": {
        "sub2": "$TEST_ROOT/test/sub_template_config2.json",
        "sub1": "$TEST_ROOT/test/sub_template_config1.json"
      }
    }
  }
}
79.4.2.2.2 OSC Sub-Template Configuration Files

sub_template_config1.json

{
  "subTemplatePath": "$FTX/templates/my_sub_template1.tosc",
  "role": "role1",
  "parameters": {
    "speed": {
      "physicalParameter": {
        "unit": "kph",
        "default": {
          "range": {
            "min": 50.0,
            "max": 60.0
          }
        },
        "generatedRecordName": "sut_gen_speed",
        "actualRecordName": "sut_actual_speed"
      }
    },
    "lead_duration": {
      "physicalParameter": {
        "validRange": {
          "min": 2.0,
          "max": 10.0
        },
        "unit": "s",
        "resolution": 0.02,
        "fixed": {
          "value": 4.0
        },
        "actualRecordName": "lead_time__actual"
      }
    },
    "slow_duration": {
      "physicalParameter": {
        "validRange": {
          "min": 2.0,
          "max": 10.0
        },
        "unit": "s",
        "resolution": 0.02,
        "generatedRecordName": "lead_time_gen"
      }
    }
  },
  "results": [
    "min_ttc",
    "max_sut_speed"
  ]
}

sub_template_config2.json

{
  "subTemplatePath": "$FTX/templates/my_sub_template2.tocs",
  "role": "role2",
  "parameters": {
    "npc_color": {
      "stringParameter": {
        "values": [
          "red",
          "blue",
          "green"
        ],
        "default": {
          "value": red"
        }
      }
    },
    "map": {
      "stringParameter": {
      }
    }
  },
  "results": [
    "min_distance",
    "max_deceleration"
  ]
}

This approach is recommended, as Sub-Template Configuration files can be reused to work with multiple top level OSC Template Configuration files for different OSC Template scenarios.

79.4.2.3 OSC Template with Sub Templates - Single File Configuration

This configuration example is for an OSC Template that uses Sub-Templates where the configuration is combined into a single file.

Instead of containing a params map for the template’s parameters, it contains a subTemplatePlaceholders that contains additional information about the subtemplates used and their specific configurations.

{
  "templatePath": "$FTX/templates/my_template.tosc",
  "maxTestDurationSeconds": 300,
  "subTemplatePlaceholders": {
    "subs": {
      "sub1": {
        "role": "role1"
      },
      "sub2": {
        "role": "role2"
      }
    },
    "subTemplateConfigs": {
      "configs": {
        "sub1": {
          "subTemplatePath": "$FTX/templates/my_sub_template1.tosc",
          "role": "role1",
          "parameters": {
            "speed": {
              "physicalParameter": {
                "validRange": {
                  "min": 20.0,
                  "max": 120.0
                },
                "unit": "kph",
                "default": {
                  "range": {
                    "min": 50.0,
                    "max": 60.0
                  }
                },
                "generatedRecordName": "sut_gen_speed",
                "actualRecordName": "sut_actual_speed"
              }
            },
            "lead_duration": {
              "physicalParameter": {
                "validRange": {
                  "min": 2.0,
                  "max": 10.0
                },
                "unit": "s",
                "resolution": 0.02,
                "fixed": {
                  "value": 4.0
                },
                "actualRecordName": "lead_time__actual"
              }
            },
            "slow_duration": {
              "physicalParameter": {
                "validRange": {
                  "min": 2.0,
                  "max": 10.0
                },
                "unit": "s",
                "resolution": 0.02,
                "generatedRecordName": "lead_time_gen"
              }
            }
          },
          "results": [
            "min_ttc",
            "max_sut_speed"
          ]
        },
        "sub2": {
          "subTemplatePath": "$FTX/templates/my_sub_template2.tosc",
          "role": "role2",
          "parameters": {
            "speed": {
              "physicalParameter": {
                "validRange": {
                  "min": 20.0,
                  "max": 120.0
                },
                "unit": "kph",
                "default": {
                  "range": {
                    "min": 50.0,
                    "max": 60.0
                  }
                },
                "generatedRecordName": "sut_gen_speed",
                "actualRecordName": "sut_actual_speed"
              }
            },
            "lead_duration": {
              "physicalParameter": {
                "validRange": {
                  "min": 2.0,
                  "max": 10.0
                },
                "unit": "s",
                "resolution": 0.02,
                "fixed": {
                  "value": 4.0
                },
                "actualRecordName": "lead_time__actual"
              }
            },
            "slow_duration": {
              "physicalParameter": {
                "validRange": {
                  "min": 2.0,
                  "max": 10.0
                },
                "unit": "s",
                "resolution": 0.02,
                "generatedRecordName": "lead_time_gen"
              }
            }
          },
          "results": [
            "min_ttc",
            "max_sut_speed"
          ]
        }
      }
    }
  }
}

79.4.2.4 Template Config Inference

The TSM library uses a mechanism to infer the configuration of the template and sub-templates based on the information provided in a loaded or provided workspace (see LoadOscTemplateConfigRequest.create). The inference mechanism differs between the two cases, as described below.

79.4.2.4.1 OSC Template Config Without Sub-Templates

In this case the inference mechanism can infer the following: * If the parameters are provided and associated with their corresponding coverage items paths, we can infer the type of the parameter, unit (if applicable) and valid values / range. * If the parameters are not provided but the coverage items paths field is defined, the relevant parameters can be infered. * If the parameters are not provided and the coverage items paths field is not defined, but the tosc file provided points to a Foretellix library scenario we can infer the parameters and their types, as long as the worksapce contains only the relevant scenario.

79.4.2.4.2 OSC Template Config With Sub-Templates

In this case the inference mechanism can fill information similarly to the previous case but also takes advantage of the 'role' attribute that can be provided in the template config files. The role is expected to be either 'config', a struct (e.g., sut.cut_in) or be left empty. The behavior for each is described below: * If the role is 'config', the inference mechanism will NOT try to infer any information. This is for configuration for templates describing map, simulator, or other configurations that don't differ much throughout the development cycle. * If the role is a struct, the inference mechanism will try to infer the parameters and their types based on the runs that affect this struct. * If the role is empty, the inference mechanism will try to infer the parameters and their types based on any of the runs in the workspace. Unexpected behavior may be encountered if there are multiple scenarios in the workspace.

79.4.2.5 FRun CSV Files

These are files that enable running a FRun job that is based on one or more CSV files. Its structure is very simple. Empty lines are ignored.

The order of the files is not important.

--csv some_main.csv
--csv some_my_config.csv
--csv some_scenarios_1.csv
--csv some_scenarios_2.csv

79.4.2.6 Accessing the Foretify Manager Python API

The Foretify Manager Python API is easily accessible from the TSM Library. You can use it for whatever functionality you need that the TSM Library doesn’t provide.

client.shell.dispatcher.get_regression_data(
    group_id="abcdef0123456789")

Alternatively, if you want to reuse existing code that already calls the Foretify Manager Python API directly, you can import the relevant modules from the TSM Library. It’s recommended to do this in the way shown below and not from a separate module. This will ensure that your code is using the same version of the Foretify Manager Python API as the TSM Library.

from python_api.tsm_client import dispatcher
dispatcher.get_regression_data(group_id="abcdef0123456789")

The two methods are functionally equivalent, with the prior not requiring another import.

79.5 API Documentation

79.5.1 TSMClient class

This is the API’s main class. It is used to connect to Foretify Manager and then perform various operations that are part of a TSM’s workflow such as running Template based Test Suites and getting their results, running FRun Test Suites, and so on.

79.5.1.1 TSMClient

Category

Constructor.

Parameters

local_work_directory (str, optional)
A local work directory where the TSM will create intermediate files. Defaults to None. This OR remote_work_directory must be provided.
remote_work_directory (str, optional)
A remote work directory where the TSM will create intermediate files. Defaults to None. This OR local_work_directory must be provided.

Note

Exactly one of "local_work_directory" or "remote_work_directory" must be provided.

All files created during a single execution of a binary using the TSM will be under a subfolder 'tsm_run_YYYY.MM.DD_hh.mm.ss_<3_digit_millis>` in the work directory.

79.5.1.1.1 Set up Network Accessible Folder With remote_work_directory

For the template scenarios to work, the API needs access to a network mapped folder that it can write files to. These files need to be accessible to Foretify Manager under that same path.

For example, if Foretify Manager has access to /ftx_data/test_files/template_inputs and we'd like to write csv files there, a local folder (on the dev machine using the installed API) by the same name needs to be mapped to that network folder. The options in this case are:

  • /ftx_data/test_files is a local folder on the dev machine and the subfolder template_inputs is mapped to the network folder /ftx_data/test_files/template_inputs.
  • /ftx_data is a local folder on the dev machine and the subfolder test_files is mapped to the network folder /ftx_data/test_files. /ftx_data local folder on the dev machine is mapped to the network folder /ftx_data.

Note

If the dev machine is a Windows machine, the local folder being mapped should be <drive>:\mapped_folder where <drive> is the one that the python script using the TSM API is running from.

79.5.1.2 TSMClient.shell

This is a class property that provides access to the underlying Foretify Manager Python API. It exposes the following interfaces.

  • dispatcher
  • test_runs
  • test_run_groups
  • workspaces

To access any of these interfaces, use the syntax TSMClient.shell.<interface>.\*.

79.5.1.3 TSMClinet.connect

A class method to connect to Foretify Manager. This must be called once at the beginning of the TSM execution before making any method calls that require use of Foretify Manager.

The method offers an optional use of a token. When using a token, the user name isn't necessary if a token file already exists from a previous login. If the token file doesn’t exist, user name is needed and a password prompt will be displayed in order for a token to be created.

Parameters

host(str)
The host address (IP or http*) where the service is available.
port(int|str)
The port number where the service is available.
user(str, optional)
The user name to use for login. Defaults to None .
use_token(bool)
Boolean flag indicating whether to use/create a JWT token. Defaults to False. If True and a new token is created, it will be saved in the user's home directory.

Returns

  • A tuple with boolean and string: Success/Fail status and an non-empty error string if the status is False (otherwise empty).

If login succeeded but token creation failed, the error message of the token creation will be returned. If both succeed, an empty string will be returned.

79.5.1.4 TSMClient.create_workspace

A static method to create a new workspace. This doesn't actually create a new workspace immediately. A workspace with the requested name will be created upon completion of the first successfully executed TSE for this workspace.

Parameters

create_workspace_request(CreateWorkspaceRequest)
A request object detailing the required information for creating a new workspace.

Returns

A CreateWorkspaceResponse object containing information about the created workspace if successful or failure information otherwise. Notice that the response.workspace_info.workspace_id will be empty until the workspace is actually created later.

79.5.1.5 TSMClient.load_workspace

A static method to load an existing workspace.

Parameters

load_workspace_request (LoadWorkspaceRequest)
A request object detailing the required information for loading an existing workspace.

Returns

  • A LoadWorkspaceResponse object containing information about the loaded workspace if successful or failure information otherwise.

79.5.1.6 TSMClient.load_osc_template_config

A static method to load an OSC Template Configuration into the TSM Client. This is required to be able to dynamically request running Test Suites with different inputs based on an OSC Template. Once such a configuration is loaded, multiple Test Suite Executions may be run with it.

Parameters

load_osc_template_config_request (LoadOscTemplateConfigRequest)
A request object detailing the required information for loading an OSC Template Configuration for a specific workspace.

Returns

  • A LoadOscTemplateConfigResponse object containing information about the loaded OSC Template Configuration if successful or failure information otherwise.

79.5.1.7 TSMClient.run_template_test_suite

This method will execute a Test Suite for a previously loaded OSC Template Configuration.

Parameters

run_template_test_suite_request (RunTemplateTestSuiteRequest)
A request object detailing the required information for running a template based Test Suite with runtime provided inputs.

Returns

Note

This includes results from the tests run in the Test Suite (per the OSC Template Configuration for which the Test Suite was requested).

79.5.1.8 TSMClient.load_frun_csv

A static method to load a FRun CSV into the TSM Client. This is required for running a Test Suite that executes tests based on existing csv files. Once such an FRun CSV is loaded, multiple Test Suite executions may be run with it.

Parameters

load_frun_csv_request (LoadFRunCSVRequest)
A request object detailing the required information for loading an FRun CSV for a specific workspace.

Returns

  • A LoadFRunCSVResponse object containing information about the loaded FRun CSV if successful or failure information otherwise.

79.5.1.9 TSMClient.run_frun_test_suite

A static method to execute a Test Suite for a previously loaded FRun CSV.

Parameters

run_frun_test_suite_request (RunFRunTestSuiteRequest)
A request object detailing the required information for running a FRun based Test Suite.

Returns

  • A RunFRunTestSuiteResponse object with information about executed Test Suite if successful or failure information otherwise.

Note

Unlike RunTemplateTestSuiteResponse, this response does not contain any results from the tests run in the Test Suite.

79.5.1.10 TSMClient.get_test_groups

A static method to get test groups information for a workspace.

Parameters

get_test_groups_request (GetTestGroupsRequest)
Details the TSM ID of the workspace (and optionally a subset of Foretify Manager Test Suite IDs) to get test groups for.

Returns

  • A GetTestGroupsResponse containing the requested test groups. Each test group details some metadata for all test runs included in the group. If an error occurred, it contains error information instead.

Note

If the workspace was created via the API and no Test Suite was successfully run with it yet, this method will fail.

79.5.1.11 TSMClient.rerun_test_groups

A static method to rerun test groups as a new Test Suite or Test Suites in a workspace.

Args

rerun_test_groups_request (RerunTestGroupsRequest)
Details the TSM ID of the workspace to rerun test groups in, along with rerun parameters for each test group.

Returns

  • A RerunTestGroupsResponse with the Foretify Manager Test Suite IDs and status for each of the Test Suites Executions. Note that it's possible to get a mix of success/failure results if more than one TSE was run.

79.5.1.12 TSMClient.get_workspace_vplan

A static method to get a workspace's VPlan.

Parameters

get_workspace_vplan_request (GetWorkspaceVPlanRequest)
Details the TSM ID of the workspace (and optionally a subset of item paths) to get VPlan for.

Returns

Note

If the workspace was created via the API and no Test Suite was successfully run with it yet, this method will fail.

79.5.1.13 TSMClient.get_test_runs_hits_info

A static method to get the buckets hits information from a workspace.

Parameters

get_test_runs_hits_info_request (GetTestRunsHitsInfoRequest)
Details the TSM ID of the workspace (required in order to retrieve the VPlan), the test suite IDs, and the test run IDs to get the information for. More optional parameters are available.

Returns

79.5.2 ResolutionRounder class

This is a utility class for rounding values to the nearest value for a given resolution. For example, Foretify requires time values to be in a resolution of 0.02 seconds. This class makes it easy to ensure time input values for OSC template parameters are in this required resolution.

The get_rounder(…) and nil_rounder() methods return cached instances, so you don’t need to persist them in your own code.

79.5.2.1 ResolutionRounder.get_rounder

A class method that returns a ResolutionRounder instance for the requested resolution.

Parameters

resolution (float | int)
The resolution that this rounder will round to. Do not pass in a value of 1. Call the nil_rounder() method instead to get a specialized instance that is more efficient.

Returns

  • A ResolutionRounder instance whose round(…) method rounds input to the nearest value in the required resolution.

79.5.2.2 ResolutionRounder.nil_rounder

A class method that returns a ResolutionRounder instance whose round(…) method returns the input value.

Returns

A specialized optimized ResolutionRounder instance whose round(…) method returns the input value.

79.5.2.3 ResolutionRounder.round

This method rounds the input number to the resolution set in this instance’s construction.

Parameters

value (float)
The number to round.

Returns

A float value that is the input rounded to the resolution set in this instance’s construction.

79.5.3 NumericRange class

This class represents an inclusive numeric range.

79.5.3.1 NumericRange.from_min_max

A class method that creates an instance of NumericRange.

Parameters

min (float | int)
The minimum value of the range.
max (float | int)
The maximum value of the range.
rounder (ResolutionRounder)
A resolution rounder to make sure values are in the required resolution. Has a default value of ResolutionRounder.nil_rounder().

Returns:

  • A NumericRange instance. Will raise an AssertionError if min > max.

79.5.3.2 NumericRange.min (property)

A float min value of the range.

79.5.3.3 NumericRange.max (property)

A float max value of the range.

79.5.3.4 NumericRange.min_max (property)

A Tuple[float, float] with min and max values of the range (in order).

79.5.3.5 NumericRange.is_in_range

A method that checks if this instance is contained in another range (inclusive).

Parameters

other_range (NumericRange)
The range that we want to check if we are in.

Returns

  • A bool result, True if this instance is contained in other_range (inclusive), otherwise False.

79.5.4 NumericValueOrRange class

This class represents either a number or a NumericRange.

79.5.4.1 NumericValueOrRange.from_value

A class method that creates an instance of NumericValueOrRange representing a simple number.

Parameters

value (float)
The numeric value.
rounder (ResolutionRounder)
A resolution rounder to make sure values are in the required resolution. Has a default value of ResolutionRounder.nil_rounder().

Returns

A NumericValueOrRange instance.

79.5.4.2 NumericValueOrRange.from_range

A class method that creates an instance of NumericValueOrRange representing a numeric range.

Parameters

range (NumericRange)
The numeric range.

Returns

  • A NumericValueOrRange instance.

79.5.4.3 NumericValueOrRange.from_min_max

A class method that creates an instance of NumericValueOrRange representing a numeric range.

Parameters

min (float | int)
The minimum value of the range.
max (float | int)
The maximum value of the range.
rounder (ResolutionRounder)
A resolution rounder to make sure values are in the required resolution. Has a default value of ResolutionRounder.nil_rounder().

Returns

A NumericValueOrRange instance. Will raise an AssertionError if min > max.

79.5.4.4 NumericValueOrRange.has_value (property)

A bool which is True iff the instance holds a number value.

79.5.4.5 NumericValueOrRange.value (property)

A float which is the number value if the instance holds a number value, otherwise None.

79.5.4.6 NumericValueOrRange.has_range (property)

A bool which is True iff the instance holds a NumericRange value.

79.5.4.7 NumericValueOrRange.range (property)

A NumericRange which is the number value if the instance holds a NumericRange value, otherwise None.

79.5.4.8 NumericValueOrRange.is_in_range

This method checks if this value or range is within a given range (inclusive).

Parameters

other_range (NumericRange)
The range that we want to check if we are in.

Returns

A bool result, True if this instance is contained in other_range (inclusive), otherwise False.

79.5.5 PhysicalTemplateParameter class

This class is used to work dynamically with OSC templates. It represents a template parameter that is used to provide a physical value. This class is not instantiated by the client in code. It is part of the OscTemplateConfig or OscSubTemplateConfig which are loaded from a JSON file or files.

79.5.5.1 PhysicalTemplateParameter.has_valid_range (property)

A bool which is True iff the parameter was assigned with a valid range. If you don't specify a valid range, the range is calculated from the VPlan.

79.5.5.2 PhysicalTemplateParameter.valid_range (property)

A NumericRange that details the range of valid values that may be given to this parameter when running a test from this template.

79.5.5.3 PhysicalTemplateParameter.set_valid_range

A method that sets the valid range of a physical parameter.

Parameters

min (float)
The minimal value of the range.
max (float)
The maximal value of the range.
rounder (ResolutionRounder)
A resolution rounder to make sure values are in the required resolution. Has a default value of ResolutionRounder.nil_rounder().

79.5.5.4 PhysicalTemplateParameter.unit (property)

A str that is used as the unit of the value. This has to be a unit that is recognized by Foretify as a valid unit type (same as the ones used in CSV files with FRun and in OSC files).

79.5.5.5 PhysicalTemplateParameter.has_resolution (property)

A bool which is True iff the parameter can accept values only in a certain resolution.

79.5.5.6 PhysicalTemplateParameter.resolution (property)

A float which is the parameter’s resolution if it has one, otherwise None.

79.5.5.7 PhysicalTemplateParameter.get_resolution_rounder

This method returns a ResolutionRounder for the resolution (if set) of the parameter. If there is no resolution, it returns the nil_rounder, which doesn't round anything.

Returns

A ResolutionRounder instance that can be used to create properly rounded values for this parameter.

79.5.5.8 PhysicalTemplateParameter.has_default_value (property)

A bool which is True iff the parameter has a default value. When trying to run a TestSuite that uses an OSC Template Config that contains this parameter, this value will be used if one wasn’t provided explicitly.

79.5.5.9 PhysicalTemplateParameter.default_value (property)

A NumericValueOrRange which is the parameter’s default value if it has one, otherwise None.

If a PhysicalTemplateParameter object has a default value, it can’t have a fixed one.

79.5.5.10 PhysicalTemplateParameter.has_fixed_value (property)

A bool which is True iff the parameter has a fixed value. When trying to run a TestSuite that uses an OSC Template Config that contains this parameter, this value will be used. If an explicit value is provided for this parameter it will be considered an error.

79.5.5.11 PhysicalTemplateParameter.fixed_value (property)

A NumericValueOrRange which is the parameter’s default value if it has one, otherwise None.

If a PhysicalTemplateParameter object has a default value, it can’t have a fixed one.

79.5.5.12 PhysicalTemplateParameter.has_implicit_value (property)

A bool which is True iff the parameter has a default value or a fixed value.

79.5.5.13 PhysicalTemplateParameter.implicit_value (property)

A NumericValueOrRange which is the parameter’s default value or fixed value if it has one, otherwise None.

79.5.5.14 PhysicalTemplateParameter.has_generated_record_name (property)

A bool which is True iff the parameter has a generated record name.

An OSC template contains parameters. The value that is passed at runtime for a parameter may be slightly different from the requested one. This is called the generated value.

An OSC template (or a file imported at any level) may contain a recording of the generated value for one or more of its parameters input value. In the output from test runs of this template, this value is recorded under a specific name. We call this the generated record name of the parameter.

Note

If the parameter has an actual record name, the actual value will be provided in test outputs in the OscTemplateTestSuiteResponse message that is returned by a call to TSMClient.run_template_test_suite.

79.5.5.15 PhysicalTemplateParameter.generated_record_name (property)

A str which is the generated record name of the parameter if it has one, otherwise None.

79.5.5.16 PhysicalTemplateParameter.has_actual_record_name (property)

A bool which is True iff the parameter has an actual record name.

An OSC template contains parameters. The value that is used at runtime may be slightly different from the generated value that was given as the parameter’s generated value. This is done to keep the scenario within defined constraints. This value at runtime is called the actual value.

An OSC template (or a file imported at any level) may contain a recording of the actual value for one or more of its parameters. In the output from test runs of this template, this value is recorded under a specific name. We call this the actual record name of the parameter.

Note

If the parameter has an actual record name, the actual value will be provided in test outputs in the OscTemplateTestSuiteResponse message that is returned by a call to TSMClient.run_template_test_suite.

79.5.5.17 PhysicalTemplateParameter.actual_record_name (property)

A str which is the actual record name of the parameter if it has one, otherwise None.

79.5.6 StringValueOrSet class

This class represents either a string or a set of strings.

79.5.6.1 StringValueOrSet.from_value

A class method that creates an instance of StringValueOrSet representing a simple string.

Parameters

value (str)
The string value.

Returns

A StringValueOrSet instance.

79.5.6.2 StringValueOrSet.from_set

A class method that creates an instance of StringValueOrSet representing a set of strings.

Parameters

string_set (str)
The set of strings.

Returns

A StringValueOrSet instance.

79.5.6.3 StringValueOrSet.has_value (property)

A bool which is True iff the instance holds a string value.

79.5.6.4 StringValueOrSet.value (property)

A str which is the string value if the instance holds a string value, otherwise None.

79.5.6.5 StringValueOrSet.has_set (property)

A bool which is True iff the instance holds a string set.

79.5.6.6 StringValueOrSet.set (property)

A Set[str] which is the set of strings if the instance holds a set, otherwise None.

79.5.7 StringTemplateParameter class

This class is used to work dynamically with OSC templates. It represents a template parameter that is used to provide a string value. This class is not instantiated by the client in code. It is part of the OscTemplateConfig or OscSubTemplateConfig which are loaded from a JSON file or files.

79.5.7.1 StringTemplateParameter.has_values (property)

A bool which is True iff the parameter was assigned with a list of valid strings, if the user doesn't specify one, the values are calculated from the VPlan.

79.5.7.2 StringTemplateParameter.values (property)

A List[str] which is the list of valid string values that this parameter can have. At least one value exists in the list.

79.5.7.3 StringTemplateParameter.allow_any_value (property)

A bool which is True iff the values list is exactly ["*"]. If True, any string value is valid for this parameter in a template run request. Note that it is invalid for a value of "*" to be one of many strings in values.

79.5.7.4 StringTemplateParameter.set_values

A method that sets the valid values of a string parameter. Calling this methods will overwrite the previous values. Note that it is invalid for a value of "*" to be one of many strings in values.

Parameters

** values (List[str])**
A list of values to set as the valid string values for this parameter.
If the input parameter equals ["*"], then any value is allowed.

79.5.7.5 StringTemplateParameter.has_default_value (property)

A bool which is True iff the parameter has a default value. When trying to run a TestSuite that uses an OSC Template Config that contains this parameter, this value will be used if one wasn’t provided explicitly.

79.5.7.6 StringTemplateParameter.default_value (property)

A StringValueOrSet which is the parameter’s default value if it has one, otherwise None.

79.5.7.7 StringTemplateParameter.has_record_name (property)

A bool which is True if this instance holds the record name for this template parameter.

79.5.7.8 StringTemplateParameter.record_name (property)

A str which is the record name for this template parameter if it exists, otherwise None.

79.5.7.9 StringTemplateParameter.has_implicit_value (property)

A bool which is True iff the parameter has an implicit value. Here it is an alias for has_default_value

79.5.7.10 StringTemplateParameter.implicit_value (property)

A StringValueOrSet which is the parameter’s implicit value if it has one, otherwise None. Here it is an alias for default_value

79.5.8 TemplateParameter class

This class configures usage of a template parameter. It provides the necessary properties for a TSM to know what values it can use for this parameter and how it should be used. It represents either a physical parameter or a string parameter. This class is not instantiated by the client in code. It is part of the OscTemplateConfig or OscSubTemplateConfig which are loaded from a JSON file or files.

79.5.8.1 TemplateParameter.has_physical_parameter (property)

A bool which is True iff this template parameter has a physical parameter value.

79.5.8.2 TemplateParameter.physical_parameter (property)

A PhysicalTemplateParameter value iff this template parameter has one, otherwise None.

79.5.8.3 TemplateParameter.has_string_parameter (property)

A bool which is True iff this template parameter has a string parameter value.

79.5.8.4 TemplateParameter.string_parameter (property)

A StringTemplateParameter value iff this template parameter has one, otherwise None.

79.5.8.5 TemplateParameter.has_implicit_value (property)

A bool which is True iff this is a physical parameter with an implicit value or a string parameter with a default value.

79.5.8.6 TemplateParameter.implicit_value_str (property)

A str with the string representation of the implicit value if it exists, otherwise None.

79.5.9 FRunCSV class

This class holds information about CSV files needed to run an FRun Test Suite.

79.5.9.1 FRunCSV.create

A class method that creates an instance of FRunCSV. Clients should not call this directly. It is returned as part of the LoadFRunCSVResponse message when calling the TSMClient.load_frun_csv method.

Parameters

frun_csv_path (str)

The path of a file containing one or more lines in the following format:

--csv /$FTX/foo/bar.csv ... --csv /baz/walso.csv

These are CSV files that will be loaded by frun to run a Test Suite. The paths may contain environment variables. Their values are as they are defined in the dispatcher job that runs the executed Test Suite.

csv_files (List[str])
A list of CSV paths read from the frun_csv_path file with the "--csv" prefix.

Returns

  • An FRunCSV instance.

79.5.9.2 FRunCSV.frun_csv_path (property)

A str with the frun_csv_path.

79.5.9.3 FRunCSV.csv_files (property)

A List[str] with the CSV paths read from the frun_csv_path file without the "--csv" prefix.

79.5.10 OscTemplateConfig class

This class is used to represent an OSC template configuration for a specific template. This configuration provides metadata about the template parameters and states what outputs are of interest when running a test based on it. See Osc Templates and Sub-Templates Configurations for an explanation of why this is necessary.

While an OSC Template Configuration is mapped to a specific template, different configurations may be used with the same template that differ in how that template is used for running Test Suites. For example, one template may set a valid range of a speed parameter dut_start_speed to [10…50]kph, while another will set it to a fixed value of 80kph.

OSC Template Configurations are loaded from JSON files (files typically end with “.json”, but this isn’t required).

This class is not instantiated by the client in code. It is part of the LoadOscTemplateConfigResponse object returned by a call to the method TSMClient.load_osc_template_config.

The methods regarding sub templates are for internal use, but are documented for completeness.

See the OSC Template Configuration Files code samples as a reference to their format.

79.5.10.1 OscTemplateConfig.template_path (property)

A str with the path of the OSC template file (“.tosc”) that this configuration is for. The path should be accessible from the dispatcher job that runs the Test Suite, and may include environment variables.

79.5.10.2 OscTemplateConfig.PhysicalParamFilter (enum)

An enumeration for filtering the returned physical parameters when calling the OscTemplateConfig.get_parameters method.

Enum values:

  • ALL - No filtering is applied to physical parameters.
  • NO_DEFAULTS - Filters out physical parameters with default or fixed values.
  • NO_FIXED - Filters out physical parameters with fixed values.
  • NONE - Filters out all physical parameters.

79.5.10.3 OscTemplateConfig.StringParamFilter (enum)

An enumeration for filtering the returned string parameters when calling the OscTemplateConfig.get_parameters method.

Enum values:

  • ALL - No filtering is applied to string parameters.
  • NO_DEFAULTS - Filters out string parameters with default values.
  • NONE - Filters out all string parameters.

79.5.10.4 OscTemplateConfig.ConfigParamFilter (enum)

An enumeration for filtering the config parameters (i.e., parameters originating from a sub-template with the role 'config') returned by the OscTemplateConfig.get_parameters method.

Enum values:

  • ALL - No filtering is applied to config parameters.
  • NO_DEFAULTS - Filters out config parameters with default values.
  • NONE - Filters out all config parameters.

79.5.10.5 OscTemplateConfig.has_parameters (property)

A bool which is True iff this instance contains parameters. If True, this means it’s a simple template that doesn’t use Sub-Templates.

79.5.10.6 OscTemplateConfig.get_parameters

This method returns a mapping from parameter names to their configuration.

Parameters

physical_parameters_filter (OscTemplateConfig.PhysicalParamFilter)
Sets a filter for the returned physical parameters. The default value is OscTemplateConfig.PhysicalParamFilter.ALL.
string_parameters_filter (OscTemplateConfig.StringParamFilter)
Sets a filter for the returned string parameters. The default value is OscTemplateConfig.StringParamFilter.ALL.
config_parameters_filter (OscTemplateConfig.ConfigParamFilter)
Sets a filter for the returned config parameters (i.e., parameters originating from a sub-template with the role 'config'). The default value is OscTemplateConfig.ConfigParamFilter.ALL.

Returns

  • A Dict[str, TemplateParameter] mapping from the parameter names to their configurations per the filtering criteria. Depending on the configuration and filtering criteria, the return value may be empty.

79.5.10.7 OscTemplateConfig.has_sub_template_placeholders (property)

A bool which is True iff this instance contains Sub Template Placeholders. If True, this Template uses Sub-Templates.

79.5.10.8 OscTemplateConfig.sub_template_placeholders (property)

A Dict[str, [TemplateSubTemplatePlaceholder] mapping from sub template placeholder names to Sub-Template Place Holders True iff this instance contains Sub Template Placeholders.

79.5.10.9 OscTemplateConfig.has_sub_template_config_json_path (property)

A bool which is True iff this instance contains Sub Template Configurations as file paths.

79.5.10.10 OscTemplateConfig.sub_template_config_json_path (property)

A Dict[str, str] mapping from Sub-Template placeholder names to Sub-Template Configuration file paths, may be empty.

79.5.10.11 OscTemplateConfig.has_sub_template_configs (property)

A bool which is True iff this instance contains Sub Template Configurations.

79.5.10.12 OscTemplateConfig.sub_template_configs (property)

A Dict[str, OscSubTemplateConfig] mapping from Sub-Template placeholder names to Sub-Template Configurations, may be empty.

79.5.10.13 OscTemplateConfig.results (property)

A List[str] with the names of all of the recorded KPIs in the OSC Template scenario that should be returned as part of the test outputs included in the RunTemplateTestSuiteResponse when calling the TSMClient.run_template_test_suite method. These names must match the record names given in the OSC Template scenario (or any nested imported scenario).

Note

OSC scenarios’ recorded values may be recorded more than once if a scenario is executed multiple times in a single test run.

Full records names are either <actor_name>.<scenario>.<record_name> if defined in a scenario, or <actor_name>.<record_name> if defined at an actor level. The <actor_name> is the name of the instance of the actor.

The names in the results are suffixes of the full name. If multiple recorded values match a suffix, only one of the values is returned. If you suspect this could be an issue for a specific scenario, provide a longer part of the full name’s suffix that will be unique to get the result you are interested in.

79.5.10.14 OscTemplateConfig.traces (property)

A List[str] with the names of all of the traces in the OSC Template scenario that should be returned as part of the test outputs included in the RunTemplateTestSuiteResponse when calling the TSMClient.run_template_test_suite method. These names must match the trace names given in the OSC Template scenario (or any nested imported scenarios).

Note

OSC scenarios’ traces may be recorded more than once if a scenario is executed multiple times in a single test run or if an actor is instantiated more than once.

Full records names are either <actor_name>.<scenario>.<trace_name> if defined in a scenario, or .<trace_name> if defined at an actor level. The <actor_name> is the name of the instance of the actor.

The names in the traces are suffixes of the full name. If multiple recorded traces match a suffix, only one of the traces is returned. If you suspect this could be an issue for a specific scenario, provide a longer part of the full name’s suffix that will be unique to get the trace you are interested in.

79.5.10.15 OscTemplateConfig.coverage_item_paths (property)

Get a map of coverage item paths. Each entry maps a coverage item path to an optional list of input parameters that affect its grade. This list may:

  • Be empty if the coverage item is not affected directly by any input parameter.
  • Contain a single parameter name if it is affected by that parameter only.
  • Contain multiple parameter names if it is affected by multiple parameters (a cross buckets item). The order of the parameters in the list should be the order of the bucket values in the item's buckets.

79.5.10.16 OscTemplateConfig.add_parameter

Parameters

name (str)
A parameter name. If there are sub-templates, the name should be in the format <sub_template_name>.<parameter_name>.
parameter (TemplateParameter)
The new template parameter.
overwrite (bool)
Optional, if True, it will overwrite the existing parameter value with the same name. If not, an existing parameter value with the same name will not be overwritten. The default value is False.

Returns

  • A bool value that is True iff the input value was added or overwritten successfully.

79.5.11 OscSubTemplateConfig class

This class is for internal use, but it is documented for completeness. You may skip this class’s documentation. This class is not instantiated by the client in code.

This class is used to represent an OSC Sub-Template Configuration for a specific sub-template. This configuration provides metadata about the sub-template parameters and states what outputs are of interest when running a test based on it. See Osc Templates and Sub-Templates Configurations for an explanation of why this is necessary.

79.5.11.1 OscSubTemplateConfig.sub_template_path (property)

A str with the path of the OSC sub-template file (“.osct”) that this configuration is for. The path should be accessible from the dispatcher job that runs the Test Suite.

79.5.11.2 OscSubTemplateConfig.has_role (property)

A bool which is True iff a role is set for this configuration.

79.5.11.3 OscSubTemplateConfig.role (property)

A str with the role if one is set, otherwise None.

79.5.11.4 OscSubTemplateConfig.parameters (property)

A Dict[str, TemplateParameter] mapping from the sub-template’s parameter names to their configurations.

79.5.11.5 OscSubTemplateConfig.results (property)

A List[str] with the names of all of the recorded KPIs in the OSC Sub-Template scenario that should be returned as part of the test outputs included in the RunTemplateTestSuiteResponse when calling the TSMClient.run_template_test_suite method. These names must match the record names given in the OSC Sub-Template scenario (or any nested imported scenario).

Note

See relevant note on the OscTemplateConfig.results method.

79.5.11.6 OscSubTemplateConfig.add_parameter

Parameters

name (str)
A parameter name.
parameter (TemplateParameter)
The new template parameter.
overwrite (bool)
Optional, if True, it will overwrite the existing parameter value with the same name. If not, an existing parameter value with the same name will not be overwritten. The default value is False.

Returns

  • A bool value that is True iff the input value was added or overwritten successfully.

79.5.12 WorkspaceInfo class

This class holds information about a workspace in the context of the TSM API. This class is not instantiated by the client in code. It is part of the CreateWorkspaceResponse and LoadWorkspaceResponse objects returned by calls to the TSMClient.create_workspace and TSMClient.load_workspace methods respectively.

79.5.12.1 WorkspaceInfo.create

This is a class method that creates a new WorkspaceInfo object.

Parameters

tsm_id (int)
The ID of the TSM. This is a unique identifier per execution of a TSM using the TSMClient Library.
workspace_id (str)
The Foretify Manager workspace ID that this TSM is related to.
workspace_name (str)
The Foretify Manager workspace name that this TSM is related to.
project_id (int)
The project ID the workspace belongs to in Foretify Manager.
project_name (str)
The project name the workspace belongs to in Foretify Manager.
timeline_point_id (str)
The timeline point ID that all TSEs will be merged into. This is the timeline point in view at the time the workspace was loaded or the first one that gets created in a new workspace.

Returns

  • A new WorkspaceInfo instance.

79.5.12.2 WorkspaceInfo.tsm_id (property)

An int ID assigned to the workspace by the TSM library.

79.5.12.3 WorkspaceInfo.workspace_id (property)

A str with the Foretify Manager ID of the workspace.

Note

When creating a new workspace, its actual creation is delayed until the first successful execution of a TSE for the workspace. Until that occurs, this value is empty.

79.5.12.4 WorkspaceInfo.workspace_name (property)

A str with the workspace’s name in Foretify Manager.

79.5.13 ResponseResult enum

An IntEnum enumeration for the result that is part of the Status attached to all responses from the TSMClient methods. It can represent a successful execution or a categorization of an error that occurred otherwise.

Enum Values:

  • UNDEFINED - An undefined result, this is not a valid value and is defined as the first value only to catch uninitialized fields of this type.
  • OK - The request was completed successfully. There might still be some errors in the response data (e.g., a test run that failed with a collision).
  • INVALID_REQUEST - The request is invalid (e.g., some required parameter is missing).
  • INVALID_TSM_ID - A request was made via the TSMClient library that requires a workspace’s TSM ID with a value that is invalid (e.g., doesn’t exist at all since such an ID wasn’t created with a call to the TSMClient.create_workspace or TSMClient.load_workspace methods).
  • INVALID_WORKSPACE_ID - The Foretify Manager workspace ID provided in some API Call is invalid.
  • DEADLINE_EXCEEDED - A timeout occurred.
  • NETWORK_ERROR - A network error occurred.
  • FMANAGER_API_FAILURE - An error was returned from a call to the Foretify Manager Python API.

79.5.14 Status class

This class represents a generic status that is part of any *Response class returned for a corresponding *Request object in any of the relevant TSMClient methods.

This class is not instantiated by the client in code.

79.5.14.1 Status.create_error_response

This is a class method that creates a new Status object representing an error.

Parameters

result (ResponseResult)
A status enum value representing the general result state.
canonical_error_code (int, optional)
An optional canonical error code if one such exists (e.g., 501 if that value was thrown by a call to Foretify Manager’s Python API).
error_message (str, optional)
An optional human readable error message that provides additional information about the error (e.g., for an INVALID_REQUEST status, will contain a message about what’s invalid).

Returns

A new Status instance containing error information.

79.5.14.2 Status.create_ok_status

A class method for creating an OK status.

Returns

  • A new Status instance representing a successful execution of a request.

79.5.14.3 Status.is_ok (property)

A bool that is True iff the response status is ResponseResult.OK.

79.5.14.4 Status.result (property)

A ResponseResult enum value.

79.5.14.5 Status.canonical_error_code (property)

An int value of a canonical error code returned from some other API call (e.g., 404, 501) if such a value exists, otherwise None.

79.5.14.6 Status.error_message (property)

A str value with a human-readable error message if there is one, otherwise None.

79.5.15 CreateWorkspaceRequest class

This class is used to create a workspace via the TSMClient.create_workspace method. Once a workspace is created, various Template Configurations and FRun CSVs can be loaded into it and then run multiple times.

79.5.15.1 CreateWorkspaceRequest.create

This is a class method that creates a new LoadWorkspaceRequest for an existing Foretify Manager workspace.

Parameters

workspace_name (str)
The name of the workspace to create.
project_id_or_name (int | str)

The ID or name of an existing project in Foretify Manager.

Note

The project must have EDITOR or OWNER access for the user credentials used to connect to Foretify Manager.

Returns

A new LoadWorkspaceRequest instance.

79.5.15.2 CreateWorkspaceRequest.workspace_name (property)

A str which is the name specified for a new Foretify Manager workspace iff the request holds such a value, otherwise None.

79.5.15.3 CreateWorkspaceRequest.has_project_id (property)

True if the request has a project ID.

79.5.15.4 CreateWorkspaceRequest.project_id (property)

An int project ID if set, otherwise None.

79.5.15.5 CreateWorkspaceRequest.has_project_name (property)

True if the request has a project name.

79.5.15.6 CreateWorkspaceRequest.project_name (property)

A str project name if set, otherwise None.

79.5.16 CreateWorkspaceResponse class

This class is used as the return value of a call to the TSMClient.create_workspace method. It contains information about the requested workspace creation. The client does not instantiate this class in code.

79.5.16.1 CreateWorkspaceResponse.from_workspace_info

This is a class method that creates a new CreateWorkspaceResponse with workspace information for a successful workspace creation.

Parameters

workspace_info (WorkspaceInfo)
The workspace information for a successful workspace creation.

Returns

A new CreateWorkspaceResponse instance.

79.5.16.2 CreateWorkspaceResponse.from_error_status

This is a class method that creates a new CreateWorkspaceResponse with an error status for an unsuccessful workspace creation.

Parameters

error_status (Status)
The error status for an unsuccessful workspace creation.

Returns

  • A new CreateWorkspaceResponse instance.

79.5.16.3 CreateWorkspaceResponse.workspace_info (property)

A WorkspaceInfo with the workspace information in case of success, otherwise None.

79.5.16.4 CreateWorkspaceResponse.status (property)

The Status of the response.

79.5.16.5 CreateWorkspaceResponse.is_ok (property)

True if the request succeeded.

79.5.17 LoadWorkspaceRequest class

This class is used to load an existing workspace via the TSMClient.load_workspace method. Once a workspace is loaded, various Template Configurations and FRun CSVs can be loaded for it and then run multiple times.

79.5.17.1 LoadWorkspaceRequest.create

This is a class method that creates a new LoadWorkspaceRequest for an existing Foretify Manager workspace.

Parameters

workspace_id (str)

The ID of an existing workspace in Foretify Manager.

Note

The workspace must exist in a project that has EDITOR or OWNER access for the user credentials used to connect to Foretify Manager. Otherwise an error will be returned.

Returns

A new LoadWorkspaceRequest instance.

79.5.17.2 LoadWorkspaceRequest.workspace_id (property)

A str which is a Foretify Manager workspace ID.

79.5.18 LoadWorkspaceResponse class

This class is used as the return value of a call to the TSMClient.load_workspace method. It contains information about the requested workspace to load. The client does not instantiate this class in code.

79.5.18.1 LoadWorkspaceResponse.workspace_info

This is a class method that creates a new LoadWorkspaceResponse with workspace information for a successful workspace to load.

Parameters

workspace_info (WorkspaceInfo)
The loaded workspace info if the request completed successfully, otherwise None

Returns

A new LoadWorkspaceResponse instance.

79.5.18.2 LoadWorkspaceResponse.from_error_status

This is a class method that creates a new CreateWorkspaceResponse with an error status for an unsuccessful workspace creation.

Parameters

error_status (Status)
The error status for an unsuccessful workspace load.

Returns

A new LoadWorkspaceResponse instance.

79.5.18.3 LoadWorkspaceResponse.workspace_info (property)

A WorkspaceInfo with the workspace information in case of success, otherwise None.

79.5.18.4 LoadWorkspaceResponse.status (property)

A Status of the requested operation for which this response was returned.

79.5.18.5 LoadWorkspaceResponse.is_ok (property)

True if the request succeeded.

79.5.19 LoadFRunCSVRequest class

This class is used to request loading a FRun CSV file to run later in a specific workspace via the TSMClient.load_frun_csv method. Once loaded successfully, a Test Suite based on this file can be run multiple times with TSMClient.run_frun_test_suite.

79.5.19.1 LoadFRunCSVRequest.create

This is a class method that creates a new LoadFRunCSVRequest.

Parameters

tsm_id (int)
The TSM ID of the workspace we want to load the FRun CSV information for. The method will raise an AssertionError if tsm_id < 1 or if it doesn’t exist.
frun_csv_path (str)
The path of the FRun CSV file to read, the format of this file is specified in the FRunCSV class. The method will raise an AssertionError if the frun_csv_path is empty.
environment_settings_name (str, optional)
The environment settings definition name in Foretify Manager to use when running Test Suites with the loaded FRun CSV. Defaults to None. This OR environment_settings_id must be provided.
environment_settings_id (str, optional)
The environment settings definition ID in Foretify Manager to use when running Test Suites with the loaded FRun CSV. Defaults to None. This OR environment_settings_name must be provided.

Returns

  • A new LoadFRunCSVRequest instance.

79.5.19.2 LoadFRunCSVRequest.tsm_id (property)

An int which is the TSM ID of the workspace we want to load the FRun CSV information for.

79.5.19.3 LoadFRunCSVRequest.frun_csv_path (property)

A str with the path of the FRun CSV file that we want to load.

79.5.19.4 LoadFRunCSVRequest.has_environment_settings_name (property)

A bool which is True iff this instance has an environment settings name.

79.5.19.5 LoadFRunCSVRequest.environment_settings_name (property)

A str with the environment settings definition name in Foretify Manager to use when running Test Suites with the loaded FRun CSV. If the instance doesn't have this value, None will be returned.

79.5.19.6 LoadFRunCSVRequest.has_environment_settings_id (property)

A bool which is True iff this instance has an environment settings ID.

79.5.19.7 LoadFRunCSVRequest.environment_settings_id (property)

A str with the environment settings definition ID in Foretify Manager to use when running Test Suites with the loaded FRun CSV. If the instance doesn't have this value, None will be returned.

79.5.20 LoadFRunCSVResponse class

This class is used as the return value of a call to the TSMClient.load_frun_csv method. It contains information about the loaded FRun CSV file. The client does not instantiate this class in code.

79.5.20.1 LoadFRunCSVResponse.create

This is a class method that creates a new LoadFRunCSVResponse.

Parameters

tsm_id (int)
The TSM ID of the workspace that the FRun CSV was loaded for.
frun_csv_id (int)
A unique ID for the loaded FRun CSV, this ID is used to request running a FRun Test Suite. The ID is unique for the duration of the binary execution.
env_settings_name (str)
The name of the environment settings definition name in Foretify Manager that will be used when running Test Suites with the loaded FRun CSV.
frun_csv (FRunCSV)
The loaded FRun CSV.

Returns

  • A new LoadFRunCSVResponse instance.

79.5.20.2 LoadFRunCSVResponse.from_error_status

This is a class method that creates a new LoadFRunCSVResponse with an error status for an unsuccessful load of a FRun CSV file. This can be caused by an error reading or parsing the FRun CSV file, or by a failure to load the environment settings from Foretify Manager.

Parameters

error_status (Status)
The error status for an unsuccessful load of an FRun CSV file.

Returns

  • A new LoadFRunCSVResponse instance.

79.5.20.3 LoadFRunCSVResponse.tsm_id (property)

The int TSM ID of the workspace that the FRun CSV was loaded for upon a successful load, otherwise None.

79.5.20.4 LoadFRunCSVResponse.frun_csv_id (property)

An int unique ID for the loaded FRun CSV upon a successful load, otherwise None. This ID is used to request running a FRun Test Suite. The ID is unique for the duration of the binary execution.

79.5.20.5 LoadFRunCSVResponse.env_settings_name (property)

A str with the name of the environment settings definition name in Foretify Manager that will be used when running Test Suites with the loaded FRun CSV upon a successful load, otherwise None.

79.5.20.6 LoadFRunCSVResponse.frun_csv (property)

The loaded FRunCSV upon a successful load, otherwise None.

79.5.20.7 LoadFRunCSVResponse.status (property)

A Status of the requested operation for which this response was returned.

79.5.20.8 LoadFRunCSVResponse.is_ok (property)

True if the request succeeded.

79.5.21 LoadOscTemplateConfigRequest class

This class is used to request loading an OCS Template Configuration to run later in a specific workspace via the TSMClient.load_osc_template_config method. Once loaded successfully, a Test Suite based on this configuration can be run multiple times with TSMClient.run_template_test_suite.

79.5.21.1 LoadOscTemplateConfigRequest.create

This is a class method that creates a new LoadOscTemplateConfigRequest.

Parameters

tsm_id (int)
The TSM ID of the workspace we want to load the OSC Template Configuration for. The method will raise an AssertionError if tsm_id < 1 or it doesn’t exist.
config_json_path (str)
The path of a json file containing an OSC Template Configuration. See example configuration files.
environment_settings_name (str, optional)
The environment settings definition name in Foretify Manager to use when running Test Suites with the loaded FRun CSV. Defaults to None. This OR environment_settings_id must be provided.
environment_settings_id (str, optional)
The environment settings definition ID in Foretify Manager to use when running Test Suites with the loaded FRun CSV. Defaults to None. This OR environment_settings_name must be provided.
filler_ws_id (str, optional)

The ID of the workspace that has a metric model. The metric model is used to infer missing information from the loaded OSC Template Configuration. If None, template config should explicitly contain all the required information. Defaults to None.

Note

The 'parameters' field is not essential if a filler_ws_id is provided. The parameters are determined based on the coverage_item_paths including their type and valid values. Units are determined from the metric model. It is important to know that the 'generatedRecordNames' field for physical parameters will remain empty.

skip_missing_values (bool, optional)
A flag to indicate whether missing valid values in the template config should be handled. If False, an error will be raised if valid values are missing for a parameter. If True filler_ws_id should not be provided. Default is False

Returns

  • A new LoadOscTemplateConfigRequest instance.

79.5.21.2 LoadOscTemplateConfigRequest.tsm_id (property)

An int which is the TSM ID that we want to load the OSC Template Configuration for.

79.5.21.3 LoadOscTemplateConfigRequest.config_json_path (property)

A str with the path of a json file containing an OSC Template Configuration.

79.5.21.4 LoadOscTemplateConfigRequest.has_environment_settings_name (property)

A bool which is True iff this instance has an environment settings name.

79.5.21.5 LoadOscTemplateConfigRequest.environment_settings_name (property)

A str with the environment settings definition name in Foretify Manager to use when running Test Suites with the loaded FRun CSV. If the instance doesn't have this value, None will be returned.

79.5.21.6 LoadOscTemplateConfigRequest.has_environment_settings_id (property)

A bool which is True iff this instance has an environment settings ID.

79.5.21.7 LoadOscTemplateConfigRequest.environment_settings_id (property)

A str with the environment settings definition ID in Foretify Manager to use when running Test Suites with the loaded FRun CSV. If the instance doesn't have this value, None will be returned.

79.5.21.8 LoadOscTemplateConfigRequest.has_filler_ws_id (property)

A bool indicating whether an explicit workspace ID was provided to infer missing information from the template config.

79.5.21.9 LoadOscTemplateConfigRequest.filler_ws_id (property)

A str of the workspace ID. If supplied, infers missing information from the template config; otherwise, it is None.

79.5.21.10 LoadOscTemplateConfigRequest.skip_missing_values (property)

A bool indicates whether to handle missing information in the template config.

79.5.22 LoadOscTemplateConfigResponse class

This class is used as the return value of a call to the TSMClient.load_osc_template_config method. It contains information about the loaded OSC Template Configuration. The client does not instantiate this class in code.

79.5.22.1 LoadOscTemplateConfigResponse.create

This is a class method that creates a new LoadOscTemplateConfigResponse.

Parameters

tsm_id (int)
The TSM ID that the FRun CSV was loaded into.
config_id (int)
A unique ID for the loaded OSC Template Configuration, this ID is used to request running a Template Test Suite. The ID is unique for the duration of the binary execution.
env_settings_name (str)
The name of the environment settings definition name in Foretify Manager that will be used when running Test Suites with the loaded OSC Template Configuration.
osc_template_config (OscTemplateConfig)
The loaded OSC Template Configuration.

Returns

  • A new LoadOscTemplateConfigResponse instance.

79.5.22.2 LoadOscTemplateConfigResponse.from_error_status

This is a class method that creates a new LoadOscTemplateConfigResponse with an error status for an unsuccessful load of an OSC Template Configuration from a json file. This can be caused by an error reading or parsing the json file, or by a failure to load the environment settings from Foretify Manager.

Parameters

error_status (Status)
The error status for an unsuccessful load of an FRun CSV file.

Returns

  • A new LoadOscTemplateConfigResponse instance.

79.5.22.3 LoadOscTemplateConfigResponse.tsm_id (property)

The int TSM ID of the workspace that the OSC Template Configuration was loaded for upon a successful load, otherwise None.

79.5.22.4 LoadOscTemplateConfigResponse.config_id (property)

An int unique ID for the loaded OSC Template Configuration upon a successful load, otherwise None. This ID is used to request running a Template Test Suite. The ID is unique for the duration of the binary execution.

79.5.22.5 LoadOscTemplateConfigResponse.env_settings_name (property)

A str with the name of the environment settings definition name in Foretify Manager that will be used when running Test Suites with the loaded OSC Template Configuration upon a successful load, otherwise None.

79.5.22.6 LoadOscTemplateConfigResponse.osc_template_config (property)

The loaded OscTemplateConfig upon a successful load, otherwise None.

79.5.22.7 LoadOscTemplateConfigResponse.status (property)

A Status with the status of the requested operation for which this response was returned.

79.5.22.8 LoadOscTemplateConfigResponse.is_ok (property)

True if the request succeeded.

79.5.23 TestId class

A test ID provided by the user of the API. The ID can be either a number or a string. Test results are returned per their user given ID. A test ID may be reused between regressions, but has to be unique within a regression.

79.5.23.1 TestId.from_string

This is a class method that creates a new TestId.

Parameters

string (str)
The string to use as a Test ID.

Returns

  • A new TestId instance.

79.5.23.2 TestId.from_number

This is a class method that creates a new TestId.

Parameters

number (int)
The number to use as a Test ID.

Returns

  • A new TestId instance.

79.5.23.3 TestId.has_string (property)

A bool which is True iff this is a string ID.

79.5.23.4 TestId.string (property)

A str which is the test ID value iff this is a string ID, otherwise None.

79.5.23.5 TestId.has_number (property)

A bool which is True iff this is a number ID.

79.5.23.6 TestId.number (property)

An int which is the test ID value iff this is a number ID, otherwise None.

79.5.24 TestRunErrorInfo class

This class is used to provide information about an error that occurred during a test run.

79.5.24.1 TestRunErrorInfo.create

This is a class method that creates a new TestRunErrorInfo.

Parameters

main_issue_result (str | None)
The main issue result of the test run.
main_issue_kind (str | None)
The main issue kind of the test run.
main_issue_details (str | None)
The main issue details of the test run.
main_issue_severity (str | None)
The main issue severity of the test run.

Returns

  • A new TestRunErrorInfo instance.

79.5.24.2 TestRunErrorInfo.has_main_issue_result (property)

A bool which is True iff this instance has a main issue result.

79.5.24.3 TestRunErrorInfo.main_issue_result (property)

A str with the main issue result of the test run.

79.5.24.4 TestRunErrorInfo.has_main_issue_kind (property)

A bool which is True iff this instance has a main issue kind.

79.5.24.5 TestRunErrorInfo.main_issue_kind (property)

A str with the main issue kind of the test run.

79.5.24.6 TestRunErrorInfo.has_main_issue_details (property)

A bool which is True iff this instance has main issue details.

79.5.24.7 TestRunErrorInfo.main_issue_details (property)

A str with the main issue details of the test run.

79.5.24.8 TestRunErrorInfo.has_main_issue_severity (property)

A bool which is True iff this instance has a main issue severity.

79.5.24.9 TestRunErrorInfo.main_issue_severity (property)

A str with the main issue severity of the test run.

79.5.25 ParameterInput class

This class is used to represent an input value of a single template parameter for a test. It can hold either a numeric value or a string.

79.5.25.1 ParameterInput.from_numeric_value

A class method that creates an instance of ParameterInput with a simple numeric value.

Parameters

value (float | int)
The numeric value input.
rounder (ResolutionRounder)
A resolution rounder to make sure values are in the required resolution. Has a default value of ResolutionRounder.nil_rounder().

Returns:

  • A ParameterInput instance.

79.5.25.2 ParameterInput.from_numeric_range

A class method that creates an instance of ParameterInput with a numeric range.

Parameters

numeric_range (NumericRange)
The numeric range input.

Returns

  • A ParameterInput instance.

79.5.25.3 ParameterInput.from_numeric_value_or_range

A class method that creates an instance of ParameterInput with a numeric value or range.

Parameters

numeric_value_or_range (NumericValueOrRange)
The numeric value or range input.

Returns

  • A ParameterInput instance.

79.5.25.4 ParameterInput.from_min_max

A class method that creates an instance of ParameterInput with a numeric range.

Parameters

min (float | int)
The minimum value of the range input.
max (float | int)
The maximum value of the range input.
rounder (ResolutionRounder)
A resolution rounder to make sure values are in the required resolution. Has a default value of ResolutionRounder.nil_rounder().

Returns

A ParameterInput instance. Will raise an AssertionError if min > max.

79.5.25.5 ParameterInput.from_string_value

A class method that creates an instance of ParameterInput with a string.

Parameters

string (str)
The string input.

Returns

  • A ParameterInput instance.

79.5.25.6 ParameterInput.from_string_set

A class method that creates an instance of ParameterInput with a set of strings.

Parameters

string_set (Set[str])
The set of strings input.

Returns

  • A ParameterInput instance.

79.5.25.7 ParameterInput.has_numeric (property)

A bool which is True iff this is a numeric input.

79.5.25.8 ParameterInput.numeric (property)

A NumericValueOrRange with the numeric input iff this is a numeric input, otherwise None.

79.5.25.9 ParameterInput.has_string (property)

A bool which is True iff this is a string input.

79.5.25.10 ParameterInput.string (property)

A StringValueOrSet with the string input iff this is a string input, otherwise None.

79.5.26 TestInput class

This class used to specify input for a single test. It includes inputs for all its required parameters and test specific attributes.

79.5.26.1 TestInput.create

This is a class method that creates a new TestInput.

Parameters

id (TestId)
A test ID, results from runs of this test will be returned with this test ID.
seed (int)
An optional seed value to use for running this test, if seed value is 0, then a random seed will be used. The default is 1.
runs (int)
An optional number of runs for this test. Multiple runs use consecutive seeds. The default is 1.

Returns

  • A new TestInput instance.

79.5.26.2 TestInput.id (property)

The TestId that the API user provided.

79.5.26.3 TestInput.parameter_values (property)

A Dict[str, ParameterInput] which maps parameter names to their inputs.

79.5.26.4 TestInput.is_random_seed (property)

A bool that is True iff this test uses a random seed.

79.5.26.5 TestInput.seed (property)

An int seed value to run this test with. If the value is 0, it uses a random seed.

79.5.26.6 TestInput.runs (property)

An int number of runs for this test. Multiple runs will use consecutive seeds.

79.5.26.7 TestInput.get_has_parameter

Parameters

name (str)
A parameter name.

Returns

A bool that is True iff the name parameter has an input value.

79.5.26.8 TestInput.get_parameter

Parameters

name (str)
A parameter name.

Returns

  • A ParameterInput for the name parameter if it has an input value, otherwise None.

79.5.26.9 TestInput.add_parameter

Parameters

name (str)
A parameter name.
value (ParameterInput)
The value to add for the parameter.
overwrite (bool)
Optional, if True will overwrite existing parameter value with the same name. If not, an existing parameter value with the same name will not be overwritten. Default value is False.

Returns

  • A bool value that is True iff the input value was added or overwritten successfully.

79.5.27 NumericParameterOutput class

This class is used to provide output values for a single physical template parameter for a test run. The client does not instantiate this class in code.

79.5.27.1 NumericParameterOutput.create

This is a class method that creates a new NumericParameterOutput.

Parameters

requested (NumericValueOrRange)
The value that was requested for this parameter (in a TestInput instance) in a test run.
generated (float | None)
An optional generated value for this template parameter in a test run. Default is None. See PhysicalTemplateParameter.generated_record_name for more details.
actual (float | None)
An optional actual value for this template parameter in a test run. Default is None. See PhysicalTemplateParameter.actual_record_name for more details.

Returns

A new NumericParameterOutput instance.

79.5.27.2 NumericParameterOutput.has_requested (property)

A bool which is True if this instance holds the requested value for this template parameter.

79.5.27.3 NumericParameterOutput.requested (property)

The NumericValueOrRange that was requested for this parameter if it exists, otherwise None.

79.5.27.4 NumericParameterOutput.has_generated (property)

A bool which is True if this instance holds the generated value for this template parameter.

79.5.27.5 NumericParameterOutput.generated (property)

A float which is the generated value for this template parameter if it exists, otherwise None.

79.5.27.6 NumericParameterOutput.has_actual (property)

A bool which is True if this instance holds the actual value for this template parameter.

79.5.27.7 NumericParameterOutput.actual (property)

A float which is the actual value for this template parameter if it exists, otherwise None.

79.5.28 StringParameterOutput class

This class is used to provide output values for a single string template parameter for a test run. The client does not instantiate this class in code.

79.5.28.1 StringParameterOutput.create

This is a class method that creates a new StringParameterOutput.

Parameters

requested (StringValueOrSet)
The value that was requested for this parameter (in a TestInput instance) in a test run.
record (str | None)
An optional recorded value for this template parameter in a test run. Default is None. See StringTemplateParameter.record_name for more details.

Returns

A new StringParameterOutput instance.

79.5.28.2 StringParameterOutput.has_requested (property)

A bool which is True if this instance holds the requested value for this template parameter.

79.5.28.3 StringParameterOutput.requested (property)

The StringValueOrSet that was requested for this parameter if it exists, otherwise None.

79.5.28.4 StringParameterOutput.has_record (property)

A bool which is True if this instance holds the recorded value for this template parameter.

79.5.28.5 StringParameterOutput.record (property)

A str which is the recorded value for this template parameter if it exists, otherwise None.

79.5.29 ParameterOutput class

Output for a single parameter provided for a test. Parameters are technically inputs to test runs, but they are of interest when looking at a test run outputs for two reasons.

  • The value of the input parameter for the test may give insight into the test run output.
  • For numerical parameters, the requested value isn't necessarily the value that will be used at test runtime. The generated value may be different from the requested value to satisfy the test's constraints, and the actual value during the test may be adjusted to satisfy the test's constraints as conditions change.
  • For string parameters, the requested value isn't necessarily the value that will be used at test runtime. If a set of strings was requested, the recorded value will be an element from the requested set that satisfies the test's constraints.

The client does not instantiate this class in code.

79.5.29.1 ParameterOutput.from_numeric_parameter_output

This is a class method that creates a new ParameterOutput for a physical template parameter.

Parameters

numeric_parameter_output (NumericParameterOutput)
A numeric parameter output.
is_explicit (bool | None)
True if the parameter was explicitly set by the user, False if it was set based on default or fixed value. Default is None which means it is not known if the value was explicitly set by the user or not.

Returns

  • A new ParameterOutput instance.

79.5.29.2 ParameterOutput.from_string_parameter_output

This is a class method that creates a new ParameterOutput for a string template parameter.

Parameters

string_parameter_output (StringParameterOutput)
A string parameter output.
is_explicit (bool | None)
True if the parameter was explicitly set by the user, False if it was set based on default value. Default is None which means it is not known if the value was explicitly set by the user or not.

Returns

  • A new ParameterOutput instance.

79.5.29.3 ParameterOutput.has_numeric (property)

A bool which is True iff this instance holds a numeric output parameter value.

79.5.29.4 ParameterOutput.numeric (property)

A NumericParameterOutput which is the numeric output parameter value if it exists, otherwise None.

79.5.29.5 ParameterOutput.has_string (property)

A bool which is True iff this instance holds a string output parameter value.

79.5.29.6 ParameterOutput.string (property)

A StringParameterOutput which is the string output parameter value if it exists, otherwise None.

79.5.29.7 ParameterOutput.has_is_explicit (property)

A bool which is True iff this instance holds an is_explicit output parameter value.

79.5.29.8 ParameterOutput.is_explicit (property)

A bool | None which is the explicitness of the parameter if known, otherwise None.

79.5.30 TestRunOutput class

This class is used to provide output for a single test run that was part of a Template Test Suite run with a call to the TSMClient.run_template_test_suite method. The client does not instantiate this class in code.

79.5.30.1 TestRunOutput.create

This is a class method that creates a new TestRunOutput.

Parameters

id (TestId)
The Test ID of the TestInput that generated the run this output is for. Multiple TestRunOutput instances for a single TestInput will have the same Test ID if the TestInput required more than one run.
Foretify Manager_test_run_id (str | None)
The test ID of this test run in Foretify Manager (if such an ID exists).
succeeded (bool)
True iff the test ran successfully (test ran to completion without any errors).
foretify_run_status (str | None)
The Foretify Status for this test run if returned from Foretify.
seed (int)
The seed used for this test run.
template_path (str)
The path to the template file used for this test run that the dispatcher used. This path will not include environment variables.
parameters (Dict[str, ParameterOutput] | None)
The output for all template parameters for this test run if there are parameters. The default is None.
results (Dict[str, float] | None)

The recorded KPIs if there are any. The default is None.

  • Keys: Record names (as specified in the [OscTemplateConfig.results][osctemplateconfigresults-property that was used to run this test).
  • Values: The recorded values.
traces (pandas.DataFrame | None)
The recorded KPIs if there are any. If traces are returned, the first column is a time-series and additional columns contain recorded values for each trace name. The trace names are as specified in the OscTemplateConfig.traces that was used to run this test. The default is None.
test_run_error_info (TestRunErrorInfo | None)
The error information for this test run if there was an error. The default is None.

Returns

  • A new TestRunOutput instance.

79.5.30.2 TestRunOutput.id (property)

A TestId with this test’s ID.

79.5.30.3 TestRunOutput.fmanager_test_run_id (property)

A str with this test’s Run ID in Foretify Manager.

79.5.30.4 TestRunOutput.succeeded (property)

A bool that is True iff the test ran successfully (test ran to completion without any errors).

79.5.30.5 TestRunOutput.foretify_run_status (property)

A str with the Foretify Status for this test run if returned from Foretify, otherwise None.

79.5.30.6 TestRunOutput.seed (property)

An int that is the seed used for this test run.

79.5.30.7 TestRunOutput.get_parameters

A Dict[str, ParameterOutput] mapping parameter names to their output. May be empty.

79.5.30.8 TestRunOutput.results (property)

A Dict[str, float] mapping result names to their output values (if they were found). May be empty.

79.5.30.9 TestRunOutput.has_traces (property)

A bool which is True iff the output contains traces from this test run.

79.5.30.10 TestRunOutput.traces (property)

A pandas.DataFrame with the traces from this test run if they exist, otherwise None.

Note

Currently not implemented.

79.5.30.11 TestRunOutput.test_run_error_info (property)

A TestRunErrorInfo with the error information for this test run if there was an error, otherwise None.

79.5.31 RunFRunTestSuiteRequest class

This class is used to request an FRun Test Suite in Foretify Manager based on an FRun CSV file (this is the equivalent of running a Test Suite in the Foretify Manager UI).

79.5.31.1 RunFRunTestSuiteRequest.create

A class method that creates an instance of RunFRunTestSuiteRequest.

Parameters

tsm_id (int)
The TSM ID to use for running the Test Suite. The Test Suite will be added to the workspace associated with this TSM. This ID must be one that was returned in a successful CreateWorkspaceResponse object (i.e. calling its status().is_ok() returns True).
frun_csv_id (int)
The ID of a FRun CSV loaded into the TSM with the TSM ID provided. This ID must be one that was returned in a successful LoadFRunCSVResponse object (i.e. calling its status().is_ok() returns True) with a matching tsm_id.

Returns

  • A RunFRunTestSuiteRequest instance.

79.5.31.2 RunFRunTestSuiteRequest.from_load_frun_csv_response

A class method that creates an instance of RunFRunTestSuiteRequest.

Parameters

load_frun_csv_response (LoadFRunCSVResponse)
The return value of a call to the TSMClient.load_frun_csv method. The TSM and FRunCSV IDs are both fetched from this input parameter. An AssertionError is raised If load_frun_csv_response.is_ok is False.

Returns

A RunFRunTestSuiteRequest instance.

79.5.31.3 RunFRunTestSuiteRequest.tsm_id (property)

An int TSM ID.

79.5.31.4 RunFRunTestSuiteRequest.frun_csv_id (property)

An int FRun CSV ID.

79.5.32 RunFRunTestSuiteResponse class

This class is the return value of a call to TSMClient.run_frun_test_suite. It contains information about the FRun Test Suite or error information.

79.5.32.1 RunFRunTestSuiteResponse.create

This is a class method that creates a new RunFRunTestSuiteResponse for a successful run of a FRun Test Suite.

Parameters

fmanager_test_suite_id (str)
The ID of the Test Suite in Foretify Manager.

Returns

  • A new RunFRunTestSuiteResponse instance.

79.5.32.2 RunFRunTestSuiteResponse.from_error_status

This is a class method that creates a new RunFRunTestSuiteResponse with an error status for an unsuccessful run of a FRun Test Suite.

Parameters

error_status (Status)
The error status for an unsuccessful run of a FRun Test Suite.

Returns

  • A new RunFRunTestSuiteResponse instance.

79.5.32.3 RunFRunTestSuiteRequest.fmanager_test_suite_id (property)

A str with the Foretify Manager Test Suite ID if it was created in Foretify Manager, otherwise an empty string. This may have a value even if the status indicates a failure.

79.5.32.4 RunFRunTestSuiteRequest.status (property)

A Status of the requested operation for which this response was returned.

79.5.32.5 RunFRunTestSuiteRequest.is_ok (property)

True if the request succeeded.

79.5.33 RunTemplateTestSuiteRequest class

This class is used to request a Template Test Suite in Foretify Manager based on an OSC Template Configuration. To use it, it must be instantiated first with one of the class methods below. After instantiation, test inputs must be added for each test we’d like to run in the Test Suite.

Note

Multiple Test Inputs with identical values for all parameters will result in only the first one being executed in the Test Suite.

79.5.33.1 RunTemplateTestSuiteRequest.create

A class method that creates an instance of RunTemplateTestSuiteRequest.

Parameters

tsm_id (int)
The TSM ID of the workspace to add the test suite to if the TSE is successful. This ID must be one that was returned in a successful CreateWorkspaceResponse or LoadWorkspaceResponse object (i.e. calling its is_ok property returns True).
config_id (int)
The ID of an OSC Template Config loaded for the workspace with the TSM ID provided. This ID must be one that was returned in a successful LoadOscTemplateConfigResponse object (i.e. calling its is_ok property returns True) with a matching tsm_id.
use_full_range_as_default (bool)
A flag that indicates whether to use the full range (physical parameters) or all possible values (string parameters) of the template parameters as the default values for the test inputs. If True, the default values for the test inputs will be the full range or all possible values of the template parameters, but only if no implicit values were provided. If False, the default values for the test inputs will be the implicit values of the template parameters. The default is False

Note

If no implicit value is provided and use_full_range_as_default is set to 'False', the cell in the frun table will remain empty. If no parameters are set, an empty test will be requested. A request cannot contain more than one empty test, and an empty test cannot be included unless it is the only test in the request. If the template config is composed of sub-templates, a line in a sub-table can be empty only if it is the only line.

Returns

  • A RunTemplateTestSuiteRequest instance.

79.5.33.2 RunTemplateTestSuiteRequest.from_load_osc_template_config_response

A class method that creates an instance of RunTemplateTestSuiteRequest.

Parameters

load_osc_template_config_response (LoadOscTemplateConfigResponse)
The return value of a call to the TSMClient.load_osc_template_config method. The TSM and config IDs are both fetched from this input parameter. An AssertionError is raised if load_osc_template_config_response.is_ok is False.
use_full_range_as_default (bool)
A flag that indicates whether to use the full range (physical parameters) or all possible values (string parameters) of the template parameters as the default values for the test inputs. If True, the default values for the test inputs will be the full range or all possible values of the template parameters, but only if no implicit was provided. If False, the default values for the test inputs will be the implicit values of the template parameters. The default is False.

Note

See note in RunTemplateTestSuiteRequest.create above.

Returns

  • A RunTemplateTestSuiteRequest instance.

79.5.33.3 RunTemplateTestSuiteRequest.tsm_id (property)

An int TSM ID of the workspace to add the TSE to after successful execution.

79.5.33.4 RunTemplateTestSuiteRequest.config_id (property)

An int config ID.

79.5.33.5 RunTemplateTestSuiteRequest.use_full_range_as_default (property)

A bool that is True iff the full range (physical parameters) or all possible values (string parameters) of the template parameters are used as the default values for the test inputs. This behavior will happen only if no other implicit values are provided for the parameter.

79.5.33.6 RunTemplateTestSuiteRequest.test_inputs (property)

A List[TestInput] of the test run inputs included in this request.

79.5.33.7 RunTemplateTestSuiteRequest.add_test_input

test_input (TestInput)
A test input to add to the list of test inputs to include in this request.

79.5.34 RunTemplateTestSuiteResponse class

A response to a RunTemplateTestSuiteRequest that contains the outputs of the requested tests.

79.5.34.1 RunTemplateTestSuiteResponse.create

Create a new response for a run of an OSC template test suite. It is possible to get a Foretify Manager ID for the test suite if it was created but failed to execute properly, in which case the is_ok property will be False, and only the Foretify Manager ID will be set.

Args

fmanager_test_suite_id (str)
The Foretify Manager test suite ID for the requested run.
test_run_outputs (List[TestRunOutput], optional)
A list of test outputs for the inputs provided in a RunTemplateTestSuiteRequest. Defaults to None.
status (Status, optional)
The status of the run template test suite request. Defaults to an OK status.

Returns

  • A new RunTemplateTestSuiteResponse instance.

79.5.34.2 RunTemplateTestSuiteResponse.from_error_status

Create a new RunTemplateTestSuiteResponse with an error status.

Parameters

status (Status)
The error status of the run template test suite request.

Returns

  • A new instance of RunTemplateTestSuiteResponse with the given error status.

79.5.34.3 RunTemplateTestSuiteResponse.fmanager_test_suite_id (property)

A str with the Foretify Manager test suite ID.

79.5.34.4 RunTemplateTestSuiteResponse.test_run_outputs (property)

A List[TestRunOutput] with test outputs for the inputs provided in the RunTemplateTestSuiteRequest.

79.5.34.5 RunTemplateTestSuiteResponse.status (property)

A Status of the requested operation for which this response was returned.

79.5.34.6 RunTemplateTestSuiteResponse.is_ok (property)

True if the request succeeded.

79.5.35 TestGroup class

A group of test runs that are separate runs of the same test. The difference between these test is the seed used, so generated and actual values of numeric parameters differ between test runs, which change the test results and outcome.

79.5.36 TestGroup.create

A class method that creates a new TestGroup.

Args

id (int)
The ID of the test group.
test_run_outputs (List[TestRunOutput])
The list of test runs outputs.

Returns - A new TestGroup.

79.5.36.1 TestGroup.id (property)

The int ID of the test group, unique to the set of parameters that define the test.

79.5.36.2 TestGroup.test_runs_outputs (property)

A List[TestRunOutput] of test runs outputs in the test group.

79.5.37 RerunTestGroupParams class

Parameters for rerunning a test group (more runs of a single test’s definition).

79.5.37.1 RerunTestGroupParams.create

A class method that creates a new RerunTestGroupParams instance.

Args

runs (intl)
The number of times to rerun the test. This has to be 1 or higher.
seed (int, optional)
The seed to use to rerun the test. A value of 0 means a random seed will be used. This has to be 0 or higher. The default is 0 (a random seed). This determines the seed of the first run. The seed of each additional run is the previous run's seed + 1.

Returns

  • A new RerunTestGroupParams instance.

79.5.37.2 RerunTestGroupParams.runs (property)

An int, the number of times to rerun the test.

79.5.37.3 RerunTestGroupParams.is_random_seed (property)

A bool, True if a random seed will be used in a rerun.

79.5.37.4 RerunTestGroupParams.seed (property)

An int, the seed to use for rerunning the test. If 0, a random seed will be used.

79.5.38 GetTestGroupsRequest class

A request to get the test groups from a workspace.

79.5.38.1 GetTestGroupsRequest.create

A class method that creates a GetTestGroupsRequest instance with an OK status.

Args

tsm_id (int)
The TSM ID of the workspace for which we want to get test groups.
fmanager_test_suite_ids (List[str])
A list of Foretify Manager TSE IDs. If not empty (the default), will only return test groups from the requested TSE IDs. The provided IDs must be in the workspace with the provided TSM ID. TSE IDs that aren't in the workspace are ignored. The default is an empty list.
only_generative_runs (bool)
A boolean flag to filter for only GENERATIVE runs (ignoring runs of types UNKNOWN, ROAD_DATA, and SMART_REPLAY). Default is False.

Note

Not supported yet, setting this parameter has no effect on the result.

Returns

  • A new instance of GetTestGroupsRequest.

79.5.38.2 GetTestGroupsRequest.tsm_id (property)

An int, the TSM ID of the workspace from which test groups should be returned.

79.5.38.3 GetTestGroupsRequest.fmanager_test_suite_ids (property)

A List[str] of Test Suite IDs. If not empty, will only fetch test groups from the specified Test Suite IDs in the workspace. If empty, will fetch the test groups from all Test Suites in the workspace.

79.5.38.4 GetTestGroupsRequest.only_generative_runs (property)

A bool value, True if the request should consider only GENERATIVE runs (ignoring runs of types UNKNOWN, ROAD_DATA, and SMART_REPLAY).

79.5.38.5 GetTestGroupsRequest.is_entire_workspace (property)

A bool value, True if the result should consider the entire workspace (no list of Test Suite IDs provided).

79.5.39 GetTestGroupsResponse class

A response to the GetTestGroupsRequest message. Contains the requested test groups or error information if an error occurred.

79.5.39.1 GetTestGroupsResponse.create

A class method that creates a GetTestGroupsResponse instance with an OK status.

Args

tsm_id (int)
The TSM ID of the workspace the test groups were fetched from.
test_groups (Dict[int, TestGroup])
Map from Test Group IDs to test groups.

Returns

  • A new instance of GetTestGroupsResponse.

79.5.39.2 GetTestGroupsResponse.from_error_status

A class method that creates a GetTestGroupsResponse instance with an error status.

Args

status (Status)
The error status of the GetTestGroupsRequest.

Returns

  • A new instance of GetTestGroupsResponse with the given error status.

79.5.39.3 GetTestGroupsResponse.tsm_id (property)

The TSM ID of the workspace the test groups were fetched from.

79.5.39.4 GetTestGroupsResponse.test_groups (property)

A Dict[int, TestGroup] mapping test group IDs to the test groups found.

79.5.39.5 GetTestGroupsResponse.status (property)

A Status of the requested operation for which this response was returned.

79.5.39.6 GetTestGroupsResponse.is_ok (property)

True if the request succeeded.

79.5.40 RerunTestGroupsRequest class

A request to rerun a list of test groups in a specified workspace. Each test group represents a single test (that can be run multiple times) and is identified by a unique ID.

79.5.40.1 RerunTestGroupsRequest.create

A class method that creates a RerunTestGroupsRequest object.

Args

tsm_id (int)
The TSM ID of the workspace to rerun the test groups in.
rerun_test_group_params (Dict[int, RerunTestGroupParams])
A map from test group IDs to rerun parameters. The IDs must be a subset of the keys in the dictionary returned from the GetTestGroupsResponse.test_groups property for the same TSM ID. Otherwise, an error will occur when calling the TSMClient.rerun_test_groups method.

Returns

  • A new RerunTestGroupsRequest instance.

79.5.40.2 RerunTestGroupsRequest.tsm_id (property)

The int TSM ID of the workspace to use to rerun the test groups.

79.5.40.3 RerunTestGroupsRequest.rerun_test_group_params (property)

A Dict[int, RerunTestGroupParams] map from Test Group IDs to test group rerun parameters.

79.5.40.4 RerunTestGroupsRequest.add_rerun_test_group_params

Add another test group rerun to the request.

Args

test_group_id (int)
The test group ID to rerun.
rerun_test_group_params (RerunTestGroupParams)
The rerun parameters for the added test.
overwrite (bool)
If False and a test group with the ID exists in the request, the method will fail and return False. if True, will overwrite existing test group's rerun parameters.

Returns

  • A bool, True if the test group to rerun was added (or rewritten) successfully.

79.5.41 RerunTestGroupsResponse class

A response to the RerunTestGroupsRequest message.

Rerunning the test groups requested by the RerunTTSE is run as a FRun Test Suite. The response contains the Foretify Manager IDs and Status for each of these Test Suites.

79.5.41.1 RerunTestGroupsResponse.create

A class method that creates a new RerunTestGroupsResponse instance with an OK status.

Args

tsm_id (int)
The TSM ID of the workspace the test groups were rerun in.
run_frun_test_suite_responses (List[RunFrunTestSuiteResponse])
A list of responses for the FRun Test Suites used to rerun the test groups.

Returns:

  • A new RerunTestGroupsResponse instance.

79.5.41.2 RerunTestGroupsResponse.from_error_status

Create a RerunTestGroupsResponse message from an error status.

Args

status (Status)
The error status to create the response from.

Returns

  • A new RerunTestGroupsResponse instance with the error status.

79.5.41.3 RerunTestGroupsResponse.tsm_id (property)

The int TSM ID of the workspace used to rerun the test groups.

79.5.41.4 RerunTestGroupsResponse.frun_test_suite_responses (property)

A List[RunFRunTestSuiteResponse] with the responses of all the FRun TSEs used to rerun the test groups.

The list of responses for the FRun Test Suites used to rerun the test groups.

79.5.41.5 RerunTestGroupsResponse.status (property)

Warning

DO NOT CALL this property. Calling this property will raise an exception. It is not supported for this response class.

To check if the response is OK for all FRun test suites responses, use the is_ok property. If False, then at least one of the FRun TSEs failed. To check which of the FRun Test Suites failed (one or more), call frun_test_suite_responses and check the is_ok and status properties of each response.

79.5.41.6 RerunTestGroupsResponse.is_ok (property)

A bool, True if the status of is_ok is True for all the contained frun_test_suite_responses statuses.

79.5.42 VPlanBucketValue class

A VPlan bucket value. May be either a range of numerical values or a string value.

79.5.42.1 VPlanBucketValue.from_range

A class method that creates a new VPlanBucketValue from a numeric range.

Args

range (NumericRange)
The numeric range to create the VPlanBucketValue from.

Returns

  • A new VPlanBucketValue.

79.5.42.2 VPlanBucketValue.from_string

A class method that creates a new VPlanBucketValue from a string.

Args

string (str)
The string to create the VPlanBucketValue from.

Returns

  • A new VPlanBucketValue.

79.5.42.3 VPlanBucketValue.has_range (property)

A bool, True if the bucket value is a numeric range.

79.5.42.4 VPlanBucketValue.range (self)

The NumericRange of the bucket value if set, or None if the bucket value is a string.

79.5.42.5 VPlanBucketValue.has_string (property)

A bool, True if the bucket value is a string.

79.5.42.6 VPlanBucketValue.string (self)

The str of the bucket value if set, or None if the bucket value is a numeric range.

79.5.43 VPlanBucket class

A VPlan Bucket. A VPlan bucket has one or more VPlan Bucket Values. It contains coverage information for a single bucket. Multiple VPlan Bucket Values are used if the bucket is used to track coverage information for a cross bucket.

79.5.43.1 VPlanBucket.create

A class method that creates a new VPlanBucket.

Args

name (str)
The name of the bucket.
hits (int)
The number of hits in the bucket.
included (bool)
True if the bucket is included (part of the grade calculation), False otherwise.
target (int)
The target number of hits in the bucket.
bucket_values (List[VPlanBucketValue])
The values in the bucket.

Returns

  • A new VPlanBucket.

79.5.43.2 VPlanBucket.name (property)

A str with the name of the bucket.

79.5.43.3 VPlanBucket.has_hits (property)

True if the bucket has hits.

79.5.43.4 VPlanBucket.hits (property)

An int with the number of hits in the bucket.

79.5.43.5 VPlanBucket.included (property)

True if the bucket is included in the grade calculation, False otherwise.

79.5.43.6 VPlanBucket.target (property)

An int with the target number of hits in the bucket.

79.5.43.7 VPlanBucket.is_cross_bucket (property)

True if the bucket is a cross- bucket (has more than one VPlanBucketValue), False otherwise.

79.5.43.8 VPlanBucket.bucket_value (property)

The VPlanBucketValue if it has a single bucket value, or None if the bucket is a cross bucket.

For a cross bucket, call bucket_values instead.

79.5.43.9 VPlanBucket.bucket_values (property)

The List[VPlanBucketValue] with the values of the bucket, at least one value should return.

This also works for a single value bucket.

79.5.43.10 VPlanBucket.add_bucket_value

Add the given bucket value to the bucket.

Args

bucket_value (VPlanBucketValue)
The bucket value to add.

79.5.43.11 VPlanBucket.add_bucket_values

Add the given bucket values to the bucket.

Args

bucket_values (List[VPlanBucketValue])
The bucket values to add.

79.5.44 VPlanItem class

A VPlan item containing coverage information for a single item. May have zero or more named buckets.

79.5.44.1 VPlanItem.create

A class method that creates a new VPlanItem.

Args

name (str)
The name of the item. Required.
is_record_item (bool)
True if the item is a record item, False otherwise.
others_bucket_hits (int, optional)
The 'others' bucket hits for the item, if there is such a count. Defaults to None.
grade (float, optional)
The grade of the item if it is graded. Defaults to None.
has_cross_buckets (bool, optional)
True if the item has cross buckets. Defaults to False.

Returns

  • A new instance of VPlanItem. Add buckets with the add_bucket or [add_buckets[vplanitemadd_buckets] methods.

79.5.44.2 VPlanItem.name (property)

A str, the name of the item.

79.5.44.3 VPlanItem.is_record_item (property)

A bool, whether the item is a record item.

79.5.44.4 VPlanItem.others_bucket_hits (property)

An int, the "others" bucket hits for the item.

This is the number of hits that did not match any of the buckets explicitly defined for this item.

79.5.44.5 VPlanItem.is_graded (property)

True if the item is graded.

79.5.44.6 VPlanItem.grade (property)

The float grade of the item if it is graded, otherwise None.

79.5.44.7 VPlanItem.has_cross_buckets (property):

True if the item has cross buckets (more than one value per bucket).

79.5.44.8 VPlanItem.add_bucket

Add a bucket to the item.

If a bucket with the same name already exists, it will be overwritten.

Args

bucket (VPlanBucket)
The bucket to add.

79.5.44.9 VPlanItem.add_buckets

Add multiple buckets to the item.

If a bucket with the same name already exists, it will be overwritten. If multiple buckets with the same name are in the list, the last one will be used.

Args

buckets (List[VPlanBucket]) - The buckets to add.

79.5.44.10 VPlanItem.get_buckets

Get the buckets of the item.

Args

unmet_targets_only (bool)
If True, only buckets with hits below their target will be returned.
included_buckets_only (bool)
If True, only buckets that are included (part of the grade calculation) will be returned.

Returns

  • A Dict[str, VPlanBucket] mapping bucket names to buckets. May be empty depending on the data and filtering parameters.

79.5.44.11 VPlanItem.get_buckets_count

Get the bucket count of the item.

Args

unmet_targets_only (bool)
If True, only buckets with hits below their target will be counted.
included_buckets_only (bool)
If True, only buckets that are included (part of the grade calculation) will be counted.

Returns

  • An int, the bucket count of the item.

79.5.45 VPlanGroup class

A VPlan group. May have zero or more VPlan items.

79.5.45.1 VPlanGroup.create

A class method that creates a new VPlanGroup.

Args

name (str)
The name of the group. Required.
grade (float, optional)
The grade of the group if it is graded. Defaults to None.

Returns

79.5.45.2 VPlanGroup.name (property)

A str, the name of the group.

79.5.45.3 VPlanGroup.is_graded (property)

True if the group is graded, otherwise False.

79.5.45.4 VPlanGroup.grade (property)

A float, the grade of the group if it is graded, otherwise None.

79.5.45.5 VPlanGroup.add_item

Add an item to the group.

If an item with the same name already exists, it will be overwritten.

Args

item (VPlanItem)
The item to add.

79.5.45.6 VPlanGroup.add_items

Add multiple items to the group.

If an item with the same name already exists, it will be overwritten. If multiple items with the same name are in the list, the last one will be used.

Args

items (List[VPlanItem])
The items to add.

79.5.45.7 VPlanGroup.items (property)

The Dict[str, VPlanItem] map from VPlan Item names to VPlan Items in the group. The returned dictionary may be empty.

79.5.46 VPlanStruct class

A VPlan struct. May have zero or more VPlan groups.

79.5.46.1 VPlanStruct.create

Create a new VPlanStruct.

Args

name (str)
The name of the struct. Required.
grade (float, optional)
The grade of the struct if it is graded. Defaults to None.

Returns

79.5.46.2 VPlanStruct.name (property)

A str, the name of the struct.

79.5.46.3 VPlanStruct.is_graded (property)

True if the struct is graded.

79.5.46.4 VPlanStruct.grade (property)

The float grade of the struct if it is graded, otherwise None.

79.5.46.5 VPlanStruct.add_group

Add a group to the struct.

If a group with the same name already exists, it will be overwritten.

Args

group (VPlanGroup)
The group to add.

79.5.46.6 VPlanStruct.add_groups

Add multiple groups to the struct.

If a group with the same name already exists, it will be overwritten. If multiple groups with the same name are in the list, the last one will be used.

Args

groups (List[VPlanGroup])
The groups to add.

79.5.46.7 VPlanStruct.groups (property)

A Dict[str, VPlanGroup] map from VPlan group names to VPlan groups of the struct. The returned dictionary may be empty.

79.5.47 VPlanSection class

A VPlan section. May have zero or more VPlan structs.

79.5.47.1 VPlanSection.create

Create a new VPlanSection.

Args

name (str)
The name of the section. Required.
grade (float, optional)
The grade of the section if it is graded. Defaults to None.

Returns

79.5.47.2 VPlanSection.name (property)

A str, the name of the section.

79.5.47.3 VPlanSection.is_graded (property)

True if the section is graded.

79.5.47.4 VPlanSection.grade (property)

The float grade of the section if it is graded, otherwise None.

79.5.47.5 VPlanSection.add_struct

Add a struct to the section.

If a struct with the same name already exists, it will be overwritten.

Args

struct (VPlanStruct)
The struct to add.

79.5.47.6 VPlanSection.add_structs

Add multiple structs to the section.

If a struct with the same name already exists, it will be overwritten. If multiple structs with the same name are in the list, the last one will be used.

Args

structs (List[VPlanStruct])
The structs to add.

79.5.47.7 VPlanSection.structs (property)

A Dict[str, VPlanStruct] map from VPlan struct names to VPlan structs of the section. The returned dictionary may be empty.

79.5.48 VPlan class

A VPlan. May have zero or more VPlan sections.

79.5.48.1 VPlan.create

A class method that creates a new VPlan.

Args

is_calculated (bool, optional)
True if the VPlan grade is calculated.
grade (float, optional)
The grade of the VPlan if it is graded. Defaults to None.

Returns

79.5.48.2 VPlan.is_graded (property)

True if the VPlan is graded.

79.5.48.3 VPlan.grade (property)

The float grade of the VPlan if it is graded, otherwise None.

79.5.48.4 VPlan.add_section

Add a section to the VPlan.

If a section with the same name already exists, it will be overwritten.

Args

section (VPlanSection)
The section to add.

79.5.48.5 VPlan.add_sections

Add multiple sections to the VPlan.

If a section with the same name already exists, it will be overwritten. If multiple sections with the same name are in the list, the last one will be used.

Args

sections (List[VPlanSection])
The sections to add.

79.5.48.6 VPlan.sections (property)

A Dict[str, VPlanSection] map from VPlan section names to VPlan Sections. The returned dictionary may be empty.

79.5.48.7 VPlan.get_item_paths_with_buckets

Get a map of full item paths to VPlanItems.

The full item path is the concatenation of the section name, struct name, group name and item name, separated by dots. Notice that struct names have two parts separated by a dot as well, i.e., section_name.struct_name_part1.struct_name_part2.group_name.item_name

Args

with_unmet_buckets_only (bool)
If True, only items with unmet buckets are returned.
with_included_buckets_only (bool)
If True, only items with included buckets (part of the grade calculation) are returned.
remove_section (bool)
If true, the section names will be removed from the returned item paths.

Returns

A Dict[str, VPlanItem] map from full item paths to VPlanItems, may be empty depending on the data and filtering options used.

79.5.48.8 VPlan.get_item_path

Get a lsit of full item paths in the VPlan.

The full item path is the concatenation of the section name, struct name, group name and item name, separated by dots. Notice that struct names have two parts separated by a dot as well, i.e., section_name.struct_name_part1.struct_name_part2.group_name.item_name

Args

remove_section (bool)
If true, the section names will be removed from the returned item paths.

Returns

A List[str] of full item paths; may be empty depending on the data.

79.5.49 GetWorkspaceVPlanRequest class

A request to get a workspace's VPlan.

79.5.49.1 GetWorkspaceVPlanRequest.create

A class method that creates a new GetWorkspaceVPlanRequest instance.

Args

tsm_id (int)
The TSM ID of the workspace to get the VPlan for.
vplan_items_paths(List[str], optional)
The paths of the VPlan items to get. Defaults to None. If None (or empty), all of the VPlan will be returned in the response.
grade_required (bool)
If True, this will ensure that the workspace VPlan score is calculated before fetching the VPlan.

Returns

  • A new GetWorkspaceVPlanRequest instance.

79.5.49.2 GetWorkspaceVPlanRequest.tsm_id (property)

The int TSM ID of the workspace to get the VPlan for.

79.5.49.3 GetWorkspaceVPlanRequest.vplan_items_paths (property)

A List[str] with the VPlan item paths to get. May be empty, in which case the returned result will only contain the entire VPlan’s grade.

See OscTemplateConfig.coverage_item_paths for details on how this can be fetched from the template configuration.

See Simple OSC Template - Single File Configuration at the end for an example of how to set it up in the template configuration (same for sub-template configurations).

79.5.50 GetWorkspaceVPlanResponse class

A response to the GetWorkspaceVPlanRequest message.

79.5.50.1 GetWorkspaceVPlanResponse.create

A class method that creates a new GetWorkspaceVPlanResponse instance with an OK status.

Args

tsm_id (int)
The TSM ID of the workspace that the VPlan belongs to.
vplan (VPlan)
The VPlan.

Returns

  • A new GetWorkspaceVPlanResponse instance.

79.5.50.2 GetWorkspaceVPlanResponse.from_error_status

Create a response message from an error status.

Args

status (Status)
The error status to create the response from.

Returns

  • A new GetWorkspaceVPlanResponse instance with the error status.

79.5.50.3 GetWorkspaceVPlanResponse.tsm_id (property)

The int TSM ID of the workspace that the VPlan belongs to.

79.5.50.4 GetWorkspaceVPlanResponse.vplan (property)

The workspace’s VPlan if the response is_ok is True, otherwise None.

79.5.50.5 GetWorkspaceVPlanResponse.status (property)

A Status of the requested operation for which this response was returned.

79.5.50.6 GetWorkspaceVPlanResponse.is_ok (property)

True if the request succeeded.

79.5.51 BucketData class

A response to the BucketData message.

79.5.51.1 BucketData.create

A class method that creates a new BucketData instance.

Args

index (int)
The index of the bucket data.
qualified_name (str)
The qualified name of the bucket.
weight (float)
The weight of the bucket calculated with respect to the VPlan.
hits (int)
The number of hits of the bucket, as appears in the VPlan.
target (int)
The target of the bucket, as it appears in the VPlan.

Returns

  • A new BucketData instance.

79.5.51.2 BucketData.index (property)

The int index of the bucket.

79.5.51.3 BucketData.qualified_name (property)

The str qualified_name of the bucket.

79.5.51.4 BucketData.weight (property)

The float weight of the bucket.

79.5.51.5 BucketData.hits (property)

The int number of hits of the bucket.

79.5.51.6 BucketData.target (property)

The int target of the bucket.

79.5.52 BucketHits class

A response to the BucketHits message.

79.5.52.1 BucketHits.create

A class method that creates a new BucketHits instance.

Args

bucket_hits (dict[int, int])
The mapping between the bucket index and the number of times the bucket is hit by a test run.

Returns

  • A new BucketHits instance.

79.5.52.2 BucketHits.bucket_hits (property)

The Dict[int, int] mapping between the bucket index and the number of times the bucket is hit by a test run.

79.5.53 TestRunHitsInfo class

A response to the TestRunHitsInfo message.

79.5.53.1 TestRunHitsInfo.create

A class method that creates a new TestRunHitsInfo instance.

Args

test_name (str)
The test name of the test run.
status (str)
The status of the test run as appears in FManager.
duration (float)
The duration of the test run.
bucket_hits (BucketHits)
The bucket hits of the test run.

Returns

  • A new TestRunHitsInfo instance.

79.5.53.2 TestRunHitsInfo.test_name (property)

The str test name of the test run.

79.5.53.3 TestRunHitsInfo.status (property)

The str status of the test run as appears in FManager.

79.5.53.4 TestRunHitsInfo.bucket_hits (property)

The BucketHits bucket hits object of the test run.

79.5.53.5 BucketData.duration (property)

The float duration of the test run.

79.5.54 GetTestRunsHitsInfoRequest class

A request to get bucket-hits information.

79.5.54.1 GetTestRunsHitsInfoRequest.create

A class method that creates a new GetTestRunsHitsInfoRequest instance.

Args

tsm_id (int)
The TSM ID of the workspace (used to get the VPlan).
test_suite_ids(List[str], optional)
The IDs of the test suites for which the bucket-hits information should be extracted. Either this or 'test_run_ids' should be provided. Defaults to None.
test_run_ids(List[str], optional)
The IDs of the test runs for which the bucket-hits information should be extracted. Either this or 'test_suite_ids' should be provided. Defaults to None.
vplan (VPlan, optional)
The VPlan to be considered to calculate the weight or importance of each bucket. If not provided, the vplan will be fetched from the workspace. Defaults to None.
weight_metric (str, optional)
The metric for calculating the vGrade as in FManager. Defaults to 'Average'. Other options are 'Binary aggregation' or 'Progressive aggregation'.
bucket_index_to_bucket_data (Dict[int, BucketData], optional)
A dictionary mapping bucket indices to their data. Defaults to None. This can be used to provide initial data for mapping buckets based on previous responses.
bucket_name_to_bucket_index (Dict[str, int], optional)
A dictionary mapping qualified names to their respective bucket indices. Defaults to None. This can be used to provide initial data for mapping buckets based on previous responses.

Returns

  • A new GetWorkspaceVPlanRequest instance.

79.5.54.2 GetWorkspaceVPlanRequest.tsm_id (property)

The int TSM ID of the workspace to get the bucket-hits information for.

79.5.54.3 GetWorkspaceVPlanRequest.test_suite_ids (property)

The list of test suite IDs from which bucket hits should be returned.

79.5.54.4 GetWorkspaceVPlanRequest.test_run_ids (property)

The list of test run IDs from which bucket hits should be returned.

79.5.54.5 GetWorkspaceVPlanRequest.vplan (property)

The VPlan to be considered to calculate the weight or importance of each bucket. If not provided, an empty VPlan will be returned here, and the vplan of the workspace will be loaded.

79.5.54.6 GetWorkspaceVPlanRequest.weight_metric (property)

The metric for calculating the vGrade as in FManager. Supported metrics by FManager are: 'Average', 'Binary aggregation', and 'Progressive aggregation'

79.5.54.7 GetWorkspaceVPlanRequest.bucket_index_to_bucket_data (property)

A dictionary mapping bucket indices (integers) to their bucket data (BucketData).

79.5.54.8 GetWorkspaceVPlanRequest.bucket_name_to_bucket_index (property)

A dictionary mapping qualified names (strings) to their respective bucket indices (integers).

Note

When requested, the bucket_index_to_bucket_data and the bucket_name_to_bucket_index properties will be transformed from an internal representation. Since the transformation is time-consuming, it is recommended to refrain from multiple unnecessary calls.

79.5.55 GetWorkspaceVPlanResponse class

A response to the GetWorkspaceVPlanRequest message.

79.5.55.1 GetTestRunsHitsInfoResponse.create

A class method that creates a new GetTestRunsHitsInfoResponse instance.

Args

tsm_id (int)
The TSM ID of the workspace (used to get the VPlan).
test_runs_hits_info(Dictstr, TestRunHitsInfo)
A dictionary detailing the hits and hit properties of the test runs.
bucket_index_to_bucket_data (Dict[int, BucketData])
A dictionary mapping bucket indices to their data.
bucket_name_to_bucket_index (Dict[str, int])
A dictionary mapping qualified names to their respective bucket indices.

Returns

  • A new GetWorkspaceVPlanResponse instance.

79.5.55.2 GetTestRunsHitsInfoResponse.from_error_status

This is a class method that creates a new GetTestRunsHitsInfoResponse with an error status for an unsuccessful call.

Parameters

error_status (Status)
The error status for an unsuccessful call.

Returns

  • A new GetTestRunsHitsInfoResponse instance.

79.5.55.3 GetTestRunsHitsInfoResponse.tsm_id (property)

The int TSM ID of the workspace to get the bucket-hits information for.

79.5.55.4 GetTestRunsHitsInfoResponse.test_runs_hits_info (property)

A dictionary mapping test run IDs (strings) to their respective TestRunHitsInfo.

79.5.55.5 GetTestRunsHitsInfoResponse.bucket_index_to_bucket_data (property)

A dictionary mapping bucket indices (integers) to their bucket data (BucketData).

79.5.55.6 GetTestRunsHitsInfoResponse.bucket_name_to_bucket_index (property)

A dictionary mapping qualified names (strings) to their respective bucket indices (integers).

Note

When requested, the bucket_index_to_bucket_data and the bucket_name_to_bucket_index properties will be transformed from an internal representation. Since the transformation is time-consuming, it is recommended to refrain from multiple unnecessary calls.