82. CI/CD User Flows (Controlled Availability)

This section describes key workflows in Foretify Manager, including creating, executing, and managing tests and test suites for CI/CD and regression pipelines. These workflows use Smart Replay to capture, replicate, and validate AV behavior across diverse scenarios, enabling scalable, structured testing within CI/CD environments.

The architecture is explicitly designed for seamless CI/CD integration. Every stage — from log ingestion to scenario replay and automated reporting — can be invoked programmatically. This enables continuous, reproducible, and scalable validation of the AV stack as part of automated build-and-test processes. Each code change is rigorously evaluated in both real-world and synthetic contexts, forming an integral part of modern DevOps workflows.

82.1 Prerequisites for CI/CD Flows

82.2 Creating and running test suites for CI/CD

Foretify Manager provides robust tools for defining and managing tests and test suites. This capability is essential for implementing Continuous Integration/Continuous Deployment (CI/CD) pipelines and executing regression test suites to ensure the reliability and safety of AV systems throughout the development lifecycle.

82.2.1 Creating tests using Smart Replay

Smart Replay allows users to capture and precisely replay specific scenarios. This functionality is valuable in verifying the AV Stack's behavior in diverse safety and edge cases, and for introducing variations to further challenge the AV Stack's capabilities.

1. Add the data to Foretify Manager:

   a. Ingest or match real world logs using the Foretify Evaluation Solution (ESR) Reference Kit

   b. Run abstract scenarios at scale. For more information, see Launching a test suite.

   c. Import scenarios from 3rd party tools.

2. Load a run or an interval into the Single Run Debugger.

3. Identify an interesting time range or interval within the run, then click Extract test. For more information, see Exporting a scenario.

   A new test is created and automatically executed within Foretify Manager with a progress bar tracking its progress.

   Note

   The new test precisely replicates the specified scenario segment, making it suitable for validating the specific scenario within CI/CD gating flow or broader regression jobs.

4. Add the data to Foretify Manager.

   a. Ingest or match real world logs using the Foretify Evaluation Solution (ESR) Reference Kit

   b. Run abstract scenarios at scale. For more information, see Launching a test suite.

   c. Import scenarios from 3rd party tools.

5. Load a run or an interval into the Single Run Debugger.

6. Identify an interesting time range or interval within the run, then click Extract test. For more information, see Exporting a scenario.

   A new test is created and automatically executed within Foretify Manager with a progress bar tracking its progress.

   Note

   The new test precisely replicates the specified scenario segment, making it suitable for validating the specific scenario within CI/CD gating flow or broader regression jobs.

   

7. Upon completion of the test extraction and execution, click the progress bar, then select Go To Run to view the results of the test extraction.

   

82.2.2 Managing tests

Once tests are available, you can organize them into a test suite. A test suite can be integrated into a CI/CD pipeline or used for broader regression jobs, such as daily or weekly runs.

The steps to create a test suite are as follows:

1. In the Foretify Manager project, click Test Management, then select Test Suites.

   

2. Click Create New or edit an existing test suite.

3. Provide a meaningful name and description.

4. Click Add test, then add relevant tests.

   

   Note that in the example above, all tests run in "exact" replication mode, ensuring consistent environmental conditions across test runs, which is ideal for CI/CD jobs.

5. Click Create.

For behavioral tests, you can define the start seed and the number of runs to execute. You have two options for the seed:

- **Manual**: Specify a fixed random seed.
- **Random**: Automatically generate a new seed each time the test suite is launched.

82.2.3 Running tests

To execute a test suite, you need to create and launch a flow definition.

A flow definition describes how a scenario or suite of scenarios is executed. In Foretify Manager, this typically involves specifying the flow type (e.g., "Run test suite", "Extract and run test"), linking environment settings (container image, cloud repository, runtime options), and defining the exact sequence of tasks or modules to be run. Flow definitions act as the technical blueprint for scenario execution: they define how each scenario is run, what environments and resources are provisioned, and the step-by-step logic to execute simulations. A flow definition can be reused multiple times.

When creating or launching a Test Plan, you specify one or more flow definitions that dictate how the scenarios listed in the plan should be executed.

82.2.3.1 Creating flow definitions

1. Navigate to the Flow Definitions page.

   

2. Click Create New.

3. Provide a meaningful name and description.

4. For CI/CD or regression flows, choose Run Test Suite as the Flow Type.

5. Select the appropriate environment settings which specify how tests are executed, including Foretify, the simulator, and the AV stack versions, as well as cloud registry location and environment variables. For more information, see Environment Settings.

6. Select the test suites to run when the flow is launched.

7. Optionally, add labels to the flow execution to facilitate easier searching and retrieval of results.

8. Click Create.

   

82.2.3.2 Running flow definitions

1. From the Flow definitions page, select the flow and click Launch.

   

   A launch model is displayed:

   

2. Click Launch to start execution immediately, or refine the flow parameters before launching. Available refinements include enabling or disabling test suites and adding or removing execution labels.

   

82.2.3.3 Managing flow execution

Clicking Launch redirects you to the Flow Executions page, where you can monitor the execution status and progress in real time.

• "Run test suite" flows comprise a single task.
• "Extract and run" flows, used for extracting and executing new tests as previously described, consist of two distinct tasks: extraction and execution.

After the execution task is complete, click the Test Suite Results link to begin analyzing the results.

82.2.4 Task logs

During flow execution, detailed task logs are generated that provide comprehensive information about each execution step. These logs are essential for debugging, monitoring, and understanding the execution process.

82.2.4.1 Accessing task logs

• Single task log: When a task generates a single log file, click the log icon in the Task Log column to view it in your browser or download it for offline analysis.

• Multiple task logs: When a task generates more than one log file, click Multiple Logs in the Task Log column to open a modal displaying all available logs. From this modal, you can:

• View individual logs by clicking the view icon next to each log entry

• Download specific logs by clicking the download icon next to each log entry.

82.2.4.2 Task log contents

Task logs contain detailed execution information including:

• Task execution progress and status updates
• Error messages and debugging information
• Performance metrics and timing data
• System-level execution details from the underlying Airflow workflow

These logs are particularly valuable for troubleshooting failed executions, optimizing performance, and maintaining audit trails for CI/CD compliance requirements.

82.3 Creating tests with variations using Smart Replay

Smart Replay is a powerful tool for generating diverse tests from existing scenarios, whether sourced from real-world data, third-party applications, or Foretellix’s abstract scenarios.

During scenario extraction you can modify parameters such as time range, actor selection, and warmup/cooldown periods to meet specific requirements. For more information, see Exporting a scenario.

After applying these modifications, the replication type transitions to behavioral, indicating that vehicles are controlled by the Foretellix driver.

To generate multiple variations, assign a unique seed to each test instance, using the actor modifier. Use the number of runs parameter to adjust the number of variations.

82.4 Admin Flow: Configuring the default Flow Definition for Single Run Debugger Extract and Run

When launching a scenario extraction from the Single Run Debugger, a preconfigured extract-and-run flow definition is used to streamline both extraction and execution. This approach ensures that users can initiate scenario extraction and testing efficiently, without needing to manually configure flow logic for each run.

To configure a default extract and run flow definition:

1. Navigate to the Flow Definitions and create a flow.
2. Select Extract and run test as the Flow Type.
3. Configure the required environment settings.

4. Enable the option to Set as default flow definition for Extract and run.

   

5. Click Create.

By default, all extraction operations initiated from the Single Run Debugger use the designated flow definition. To override the default, navigate to the extract test settings page and select a different flow definition.

82.5 Sharing Flow Definitions and Test Suites

You can share flow definitions and test suites with other users by setting permission levels (VIEWER, EDITOR, OWNER) for specific users or groups.

82.5.1 How to Share Flow Definitions and Test Suites

1. Navigate to a Foretify Manager project, then click Test Management and open the flow definition or test suite you want to share.

2. Click the Share icon in the header.

   

3. In the Share dialog:

   • Select users and groups, and assign permission levels.
   • To enable organization-wide sharing, toggle Enable general access to the entire organization, including new users.
   • Click Copy link to share the direct URL.

   

   • Click Done to save.