74. Foretify Manager API (Python SDK)

Foretify Manager's Python SDK (Software Development Kit) allows you to interact with Foretify Manager's API. For example, you can extract data from a test suite and use it to create reports, graphs and other project-tracking materials. Foretify Manager’s Export to CSV feature offers a basic capability, but if that does not meet your needs, you can create Python scripts that incorporate test suite data into a format of your choosing. You can then execute these scripts from Foretify Manager as a user command (action).

The SDK provides, in the ftx namespace, a few modules:

• shell: abstract APIs to simplify the interaction with Foretify Manager (as explained in detail in Foretify Manager SDK APIs below)
• clients: concrete wrappers to the HTTP API exposed by Foretify Manager
• model: helper builder classes to assist with creating complex objects, such as Filters.

74.1 Installation

The Python SDK needs to be installed in your Python development environment. It is available in a .whl ("wheel") package format, located in the /client/sdk directory of the Foretify Manager distribution package.

The wheel package is versioned, so its filename (and internal metadata) specifies the Foretify Manager's server version whose API it matches, and the Python version it was built with: fmanager_python_sdk-[server_version]-[python_version]-none-any.whl (for example, fmanager_python_sdk-22.12.0.4-py3-none-any.whl).

For the user's convenience, a permanent symlink named fmanager_python_sdk-ver-py3-none-any.whl is placed at the same directory, pointing to the versioned filename.

To install Foretify Manager’s Python SDK wheel into your environment, run:

pip install "$(readlink -f fmanager_python_sdk-ver-py3-none-any.whl)"

Additional dependencies need to be specified when installing the wheel.

For example, to install Foretify Manager’s Python SDK wheel with AWS S3 and Okta, run:

pip install "$(readlink -f fmanager_python_sdk-ver-py3-none-any.whl)[s3,okta]"

74.2 Authenticating and Managing Credentials

Logging into Foretify Manager requires calling the login API with username/password credentials or an access token.

74.2.1 Using Access Tokens

Credentials are frequently used by scripts to authenticate with the server. Instead of providing the actual username/password credentials, it is possible to generate a user-specific access token, which can only be used to authenticate with Foretify Manager (even if the credentials are managed by LDAP or some Single Sign-On service).

Access tokens can also be invalidated after they are created, either by setting an expiration period during their generation (unlimited by default), or by specifically requesting to invalidate them.

Multiple access tokens can be created for each user.

For detailed information about creating and invalidating access tokens, see Access Tokens API below.

74.2.2 Managing Credentials

Instead of typing a password when prompted, you can store the connection details (such as a password or access token) in a hidden file located in the $HOME directory, which is protected from being read by other users on this machine by default.

After creation of this file, Python scripts can be executed without prompting for the user's password.

Connection details can be added by using the add_credentials utility, or manually by the following steps:

1. Create the file named $HOME/.ftxpass:

   vi $HOME/.ftxpass
   

   or

   gedit $HOME/.ftxpass
   

2. The content of the file should be a JSON object with a field named credentials, containing a list of objects of the form {"host": string, "port": string, "https": boolean, "user": string, "password": string, "access_token": string} as shown in the following example:

   {
       "credentials": [
           {
               "host": "fmanager-prod.domain.com",
               "port": "8080",
               "user": "a_user",
               "password": "a_password"
           },
           {
               "host": "fmanager-prod.domain.com",
               "port": "8080",
               "user": "some_user",
               "access_token": "some_token"
           },
           {
               "host": "fmanager-qa.domain.com",
               "port": "443",
               "https": true,
               "user": "test_user",
               "access_token": "test_token"
           }
       ]
   }
   

   Notes

   • The list can contain multiple credentials for any hostname, so the first one from the top that matches the required hostname (and username, if provided) will be used.
   • The https field is optional and is set to false by default.
   • You must set either a password or a access_token. Only one is required.

3. Make sure this file is only readable by your OS user:

   sudo chmod 600 $HOME/.ftxpass
   

4. Set the user authentication required to upload run results from the $HOME/foretify/runs directory to the Foretify Manager server for analysis and visualization:

   • Without user authentication, i.e., no password required:

     upload_runs --runs_top_dir $HOME/foretify/runs
     

   • With user authentication:

     upload_runs --runs_top_dir $HOME/foretify/runs --user some_user
     

     If only a username was provided but an appropriate password wasn't found in the .ftxpass file, the script will prompt the user to type the password.

   For more information on uploading runs, see upload_runs.

74.2.3 Session Management Utilities (Admin/Support Only)

invalidate_sessions - Command-line utility to invalidate user sessions

This utility allows administrators and support staff to forcibly terminate user sessions and release associated licenses from the command line.

Usage:

# Invalidate all sessions for a specific user
invalidate_sessions --user admin@fmanager.com --target_user john.doe \
                   --reason "Security incident"

# Invalidate a specific session by ID
invalidate_sessions --user admin@fmanager.com --session-id A1B2C3D4E5F6 \
                   --reason "Stuck session cleanup"

# With HTTPS and custom port
invalidate_sessions --user admin@fmanager.com --hostname manager.company.com \
                   --port 8443 --https --target_user john.doe

Options:

*Either --session-id OR --target_user must be provided (mutually exclusive)

Getting Session IDs:

Use the list_users utility (also admin/support only) to view active sessions:

list_users --user admin@fmanager.com

This displays all logged-in users with their session counts and license consumption.

74.3 Foretify Manager SDK Entities

The SDK provides several entities that represent aspects of a full verification environment.

Singular entities are Python dictionaries (dict), so specific attributes are accessed using the square brackets. For example, the name of a Test Run Group element trg is provided by trg['name'].

On the other hand, multiple entities (e.g. the result of page() or get() calls) are almost always a Pandas DataFrame, which is essentially a table of entities (and not a list). For example, on a given DataFrame of Test Runs df:

• A collection of all the IDs is provided by df['id'].
• The first Test Run (as a DataFrame with 1 row, not a dict) is provided by df[0:1].
• The first two are provided by df[0:2].
• mainIssueKinds of the first two run are provided by df['mainIssueKind'][0:2].

74.3.1 Metric Model

A metric model is a representation of metrics domain (e.g. coverage items and KPIs) which was defined for a given group of test runs. A metric model is defined by using OpenScenario2.0 constructs (cover() or record()) as part of scenario definition.

74.3.1.1 Struct model

74.3.1.2 Interval Model

An Interval model is a detailed representation of the structure of the intervals in this TSR. Interval models can have multiple types that correspond to the interval types.

74.3.1.3 Watcher Model

74.3.1.4 Group Model

74.3.1.5 Item

74.3.1.6 Bucket

74.3.1.7 Scenario Model

74.3.1.8 Global Modifier Model

74.3.2 VPlan Template

A VPlan template is a verification plan that facilitates analysis by organizing metrics and run data into a hierarchical framework.

74.3.2.1 VPlan Template Section

A VPlan template Section is a node in the VPlan tree hierarchy, containing coverage metrics and possibly additional nested sections.

74.3.2.2 VPlan Template Checker

A VPlan template Checker is a node in the VPlan tree hierarchy, used to contain KPIs.

74.3.2.3 VPlan Template Reference

A VPlan template Reference is a node in the VPlan tree hierarchy, used to mirror (for the purpose of re-use) another section in its entirety (either from the same VPlan template or from a different one).

74.3.3 Test Run Group

A Test Run Group, also referred to as Regression, is a representation of a group of Foretify runs, together with their corresponding metric models and some meta-data (as described below).

74.3.4 Test Run

A Test Run is a representation of a single Foretify run, saved in Foretify Manager.

74.3.4.1 Test Run Issue

A Test Run Issue is a representation of a single issue in a run.

74.3.4.2 Test Run Bucket Summary

A Test Run Bucket Summary is a representation of a single cover/record|kpi bucket hit (in case buckets for it are defined in OpenScenario2).

74.3.4.3 Test Run Group Sample

A Test Run Group Sample is a detailed representation of the sampled items in a run.

74.3.4.4 Test Run Interval

A Test Run Interval is a detailed representation of the occurrences and events in a run. An interval has a start time and an end time and hence a duration. Intervals can have multiple types and relations between them.

74.3.4.4.1 Scenario Interval

A representation of a scenario execution during simulation

74.3.4.4.2 Coverage Interval

A representation of a coverage sampling event during simulation

74.3.4.4.3 Watcher Interval

A representation of a watcher sampling event during simulation

74.3.4.4.4 Matcher Interval

A representation of a scenario match sample detected by the Evaluation Pipeline.

74.3.4.4.5 Anomaly Interval

A representation of an anomaly interval detected by the Evaluation Pipeline.

74.3.4.4.6 Behavior Interval

A representation of a Behavior interval.

74.3.4.4.7 Global Modifier Interval

A representation of a Global Modifier interval.

74.3.5 Project

A Project is a representation of a V&V project in Foretify Manager. An admin can create projects and allow specific users and user groups to access it.

74.3.6 Attribute

Attribute is a representation of a test run field in Foretify Manager. Attributes can be created from Foretify and Foretify Manager.

74.3.7 Workspace

A Workspace is a representation of a Foretify Manager Workspace, which allows annotating a VPlan template with coverage grades in the context of one or more Test Run Groups.

74.3.8 WorkspaceSection

A WorkspaceSection is a node in the workspace tree hierarchy, retrieved by applying a VPlan template section to some timeline point.

74.3.9 Struct

A Struct is a metric group or scenario grouping within a WorkspaceSection.

74.3.10 Group

A Group is a collection of items within a struct, used for organizing metrics.

74.3.11 Item

An Item is a metric or coverage point within a group.

74.3.12 Bucket

A Bucket is a container for hit counts and targets within an item.

74.3.13 VplanTemplateInclusionFilter

A filter for including or excluding runs/items in a section based on attributes or criteria.

74.3.14 VplanAttributeValue

A custom attribute and its value assigned to a section.

74.3.15 RunIssueCount

Tracks issue statistics (e.g., errors, warnings) for a section, struct, group, or item.

74.3.16 NumericAggregationTypeE

Type of aggregation method for grading.

74.3.17 TargetEditLevel

Level at which a target was edited.

74.3.18 Severity

Severity of an issue.

74.3.19 Status

Status of a test run.

74.3.20 Workspace Grading

A workspace grading object represents the workspace's VGrade, for various grading schemes.

74.3.21 Timeline Point

A timeline point is a view of a test suite in the context of a workspace. It can be constructed from one or more raw test suites. A timeline point also contains coverage data information (such as VGrade) which is workspace-specific.

74.3.22 Triage Comparison Settings

The Triage comparison settings are an object attached to a workspace, describing how test suites are compared in the workspace, in a triage context.

74.3.23 Triage View

Represents a view through which one can perform triage operations via his/her workspace. The view determines the way runs are clustered and shown in a triage context.

74.3.24 Triage Rule

Represents a triage rule. A rule can be appliedin a context of a triage, in order to update all runs in a workspace which match some filter.

74.3.25 Compound Rule

A compound rule combines two sets of intervals using a temporal relation and an action. A compound rule can be applied in a workspace or globally to create a new interval based on a relationship between two sets of intervals.

74.3.26 Compound Rule Definition

This defines the core logic of the compound rule, which combines interval filters and applies a temporal relation and action to create a new interval.

74.3.27 Temporal Relation Parameters

Defines the filters, time relation, and custom time relation used to specify the relationship between two sets of intervals.

74.3.28 Custom Time Relation

Defines specific relationships between intervals, such as the relation between their start and end points. This enables more advanced control over how intervals interact. All times are given in milliseconds.

74.3.29 Open Range

Defines a range with upper and lower limits (bound), used in the CustomTimeRelation to specify the valid time range for the interval relationship.

74.3.30 Time Relation Enum

Define different types of temporal relationships between intervals, describing how they relate to one another in terms of time.

74.3.31 CompoundCreationContext Enum

Defines the context in which the compound rule is created, either globally or within a specific workspace.

74.3.32 Dispatcher service

Dispatcher is a service, used in order to run foretify on scale. It is responsible for managing test run group definitions, and using them to execute multiple test runs in a distributed environment.

74.3.33 Test Run Group Definition

A configuration used in order to execute a test suite via dispatcher.

74.3.34 Dispatcher Environment Settings

A part of the test run group definition, which contains the environment-specific settings for running foretify (such as docker images for foretify/sut, number of cores, and so on...). Number of fields is large, and can change between environment, and thus, the configuration is a generic object, which sits under the "settings" field.

74.3.35 Foretify Manager server exception

An exception that is raised when the Foretify Manager Server encountered an error.

74.3.36 Session Invalidation Response

Response object returned from session invalidation operations (admin/support only).

74.3.37 Invalidated Session Details

Details of a single invalidated session.

74.4 Foretify Manager SDK APIs

74.4.1 Common Parameter Models

Some parameters appear in multiple endpoints of the API:

• Filter is a multi-level set of logical conditions. It is usually easiest to generate by helper methods like Filter.all() or Filter.any(). See detailed information about Filters below.

• Pagination is a dictionary object of the format {"size": [Integer], "page": [Integer]}, specifying the max number of entities which is expected in the result, along with the page number. For example, the first 50 results will be returned when specifying pagination={"size": 50, "page": 0}, the next 50 results will be returned by pagination={"size": 50, "page": 1}, etc. Requesting multiple entities from the server is always done in pages by calling page() methods, as a best practice, in order to optimally utilize network resources. Helper get() methods contained in all of the clients (test_runs.get(), test_run_groups.get(), etc.) can help abstracting the iteration and concatenation of all pages into a single call.

• OrderBy is a dictionary object of the format {"properties": ["property1", "property2", ...], "direction": "DIRECTION"}. The properties field is a list of property names (one or more) to sort the results by. The direction field specifies whether the results should be ordered by ascending ("ASC") or descending ("DESC") order. For example, the results will be sorted by time of creation, starting from the newest, by specifying orderBy={"properties": ["created_at"], "direction": "DESC"}.

74.4.2 Filters

Filters are objects used when querying for any entity (e.g. when calling page() or get() APIs). The search module provides API for creating filters from expressions.

A filter is always made up of one or more logical conditions, where all (AND) or any (OR) of the conditions need to be met.

In addition, a filter can be "inclusive" ("in-filter", i.e. passing the filter is required be included in the result) or an "exclusive" ("out-filter", i.e. not passing the filter is required be included in the result).

74.4.2.1 Conditions

A condition can be created from a string, specifying the entity's field to consider, a conditional operator on it and values to compare to.

Examples:

runs_condition_1 = Condition.from_term("testRunGroupId EQ " + trg_id)
runs_condition_2 = Condition.from_term("seed LTE 4")
trg_condition_1 = Condition.from_term("name MATCHES *nightly_regression*")
trg_condition_2 = Condition.from_term("passed GTE 1000")

The following condition operators are supported:

• EQ (equals)
• MATCHES (using wildcard pattern matching)
• NOT_MATCHES (using wildcard pattern matching)
• GT (greater than)
• GTE (greater than or equals)
• LT (less than)
• LTE (less than or equals)
• CONTAINS_ALL (contains all elements of a list)
• CONTAINS_ANY (contains at least one element in a list)
• CONTAINS_NONE (contains none of the elements in a list)
• BUCKET_NAME_EQUALS (for ItemFilter)

74.4.2.2 Filter, RunFilter, ItemFilter, IntervalFilter & AdvancedIntervalFilter

A filter can be created from one or conditions by either the Filter.all() or Filter.any() builder methods, as described below:

Any API which requires a filter will accept a basic filter created by Filter.all() or Filter.any() (which by default resolve to what is described below as a RunFilter). However, sometimes more complex filters are required. For example, it might be required to specify an exclusive filter (by setting include=False) or to create an ItemFilter to query only for runs for which meet some metric-related attributes conditions:

74.4.2.3 FilterContainer

A FilterContainer can be created from one or more Filter (interpreted as inclusive RunFilters), RunFilters, ItemFilters or IntervalFilter interchangeably.

Note: All page() and get() APIs described below expect either: - a single RunFilter (or the simpler Filter), or - a FilterContainer which contains only RunFilters.

Note: ItemFilter and IntervalFilter should only be contained in a Workspace's runsFilter attribute. In all other cases (i.e. outside the context of a Workspace), including an ItemFilter or an IntervalFilter inside a FilterContainer passed to page() or get() calls might return an empty result.

Example:

from ftx.model.search import Filter, RunFilter, ItemFilter, FilterContainer, Condition, LogicalOperator, IntervalFilter
from ftx.shell import client

client.login(host=host, user=user, port=port, https=https)

condition = Condition.from_term("passed GTE 1000")
ws_filter = FilterContainer.of(
  RunFilter.of(Filter.any("testRunGroupId EQ " + trg1_id, "testRunGroupId EQ " + trg2_id), include=True),
  RunFilter.of(Filter.single("testName EQ MyInterestingTest"), include=True),
  RunFilter.of(Filter.any("simulationTime GTE 5000", "testMap everywhereButThere"), include=False),
  RunFilter.of({"logicalOperator": LogicalOperator.AND, "conditions": [condition]}),
  ItemFilter.of(group_qualified_name="vehicle.drive.end", 
    filter=Filter.any(
      "other_vehicle_side_of_collision_while_self BUCKET_NAME_EQUALS front_right",
      "other_vehicle_side_of_collision_while_self BUCKET_NAME_EQUALS back_left"
    )
  ),
  IntervalFilter.of(
    interval_name="vehicle.drive.end", 
    interval_type=IntervalFilter.IntervalType.CoverageIntervalData, 
    filter=Filter.single("duration eq 1000")
  )
)

74.4.3 Client API

Provides methods to log in to and logout from a Foretify Manager server.

74.4.3.1 Access Tokens API

Access Tokens can be created by users to be used later for authentication, as an alternative to user/password credentials.

Example:

from ftx.shell import client

client.login(host=host, user=user, port=port, https=https)

# Generate an un-expiring token
token = client.generate_token(alias="my 1st token")["jwtToken"]

# Logout from current session, before logging-in with the token
client.logout()

# Start a new session using the token
client.login(host=host, access_token=token)

# Generate a second token
token = client.generate_token(alias="my 2nd token")["jwtToken"]

# Token is invalidated, but session is still active
client.invalidate_token(token)

# Logout from session
client.logout()

# Running this expression will result in an authentication error, since the token was invalidated
# client.login(host=host, access_token=token)

74.4.4 Users API

Provides methods for user management and session administration.

74.4.4.1 Session Management (Admin/Support Only)

IMPORTANT: The following functions require ADMIN, FMANAGER_ADMIN, or FMANAGER_SUPPORT roles.

Example:

from ftx.shell import client
from ftx.shell import users

# Login as admin or support user
client.login(host="localhost", user="admin@fmanager.com", port=8080)

# Get list of logged-in users
logged_users = users.get_logged_in_users()
for user_info in logged_users:
    print(f"User: {user_info['userName']}, "
          f"Sessions: {user_info['loggedInCount']}, "
          f"Licenses: {user_info['consumedLicensesCount']}")

# Invalidate all sessions for a specific user
response = users.invalidate_sessions(
    username="john.doe",
    reason="Security incident - suspected compromised account"
)

print(f"Invalidated {response['invalidatedCount']} session(s)")
for session in response['invalidatedSessions']:
    print(f"  - Session {session['sessionId']}: {session['sessionDuration']}, "
          f"License: {session['hadLicense']}")

# Invalidate a specific session by ID
# (Get session ID from get_logged_in_users response)
response = users.invalidate_sessions(
    session_id="A1B2C3D4E5F6",
    reason="Stuck session cleanup"
)

client.logout()

74.4.5 Metric Models API

Provides methods for uploading metric models from disk to the Foretify Manager server and for metric models analysis.

Examples:

from ftx.shell import client, metric_models, workspaces

client.login(host=host, user=user, port=port, https=https)

workspace = workspaces.get_by_id(ws_id)
model = metric_models.get_by_id(workspace["metricModelId"])

from ftx.shell import client, metric_models, workspaces

client.login(host=host, user=user, port=port, https=https)

workspace = workspaces.get_by_id(ws_id)
model = metric_models.get_by_id(workspace["metricModelId"])
buckets = list()
for struct in list(model["structs"].values()):
  for group in list(struct["groups"].values()):
    for item in list(group["items"].values()):
      for bucket in list(item["buckets"].values()):
        buckets.append(bucket)

74.4.6 VPlan Templates API

Provides methods for uploading VPlan template files to Foretify Manager server and for their analysis.

Notes:

VPlan Template Create Request is a python dictionary of the following structure:

{
  "name": "String",
  "sections": [],
  "description": "String",
  "requirementsProjectId": "String",
  "source": "String",
  "workspaceId": "String",
  "excludedVplanPaths": [],
}

All the fields except name and sections are optional.

It is advised not to manually write (or code) a VPlan template. Instead, use the VPlan Editor of the Foretify Manager web-client, and download it as a JSON file if needed. See Creating VPlans for more details.

Examples:

from ftx.shell import client, vplan_templates

client.login(host=host, user=user, port=port, https=https)

vplan_template = vplan_templates.collect("./demo/demo.vplan")
vplan_template = vplan_templates.create(vplan_template) 

from argparse import ArgumentParser
from ftx._common.utils import add_common_arguments
from ftx.shell import client, vplan_templates

def parse():
    parser = ArgumentParser(description="Arguments for add/edit VPlan sections example")
    add_common_arguments(parser)
    return parser.parse_args()

args = parse()
globals().update(vars(args))

client.login(host=host, user=user, port=port, https=https)

vplan_template = vplan_templates.create(vplan_templates.collect("./demo/demo.vplan"))
vplan_id = vplan_template["id"]


from ftx.shell import client, vplan_templates

client.login(host=host, user=user, port=port, https=https)

vplan_template = vplan_templates.get_by_id(vplan_id)

# Add a new regular section under 'top' section
new_section = {
    "name": "new_section",
    "description": "A new section under 'top' section",
    "weight": 5
}
vplan_template = vplan_templates.add_new_section(
    vplan_template_id=vplan_id,
    parent_section_path=["top"],
    new_section=new_section
)
                                              # top -> new_section
section_to_update = vplan_template["sections"][0]["sections"][1]
section_to_update["name"] = "new_section_name"
section_to_update["description"] = "Updated description for 'new_section'"
section_to_update["weight"] = 10

vplan_template = vplan_templates.update_section(
    vplan_template_id=vplan_id,
    section_path=["top", "new_section"],
    updated_section=section_to_update
)

# Add new reference section
new_reference_section = {
    "_type": "VplanTemplateReferenceRequest",
    "name": "reference_section",
    "vplanRefId": vplan_id,
    "vplanRefPath": ["top", "drive_section"]
}

vplan_template = vplan_templates.add_new_section(
    vplan_template_id=vplan_id,
    parent_section_path=["top"],
    new_section=new_reference_section
)

# Edit the reference section
                                              # top -> reference_section
section_to_update = vplan_template["sections"][0]["sections"][2]
section_to_update["name"] = "reference_section_updated"
section_to_update["vplanRefPath"] = ["top", "new_section_name"]
# can also change section_top_update["vplanRefId"] to refer to section from a different vplan template

vplan_template = vplan_templates.update_section(
    vplan_template_id=vplan_id,
    section_path=["top", "reference_section"],
    updated_section=section_to_update
)

# Delete the newly added section
vplan_template = vplan_templates.delete_section(
    vplan_template_id=vplan_id,
    section_path=["top", "new_section_name"]
)

# Delete the newly added reference section
vplan_template = vplan_templates.delete_section(
    vplan_template_id=vplan_id,
    section_path=["top", "reference_section_updated"]
)

from ftx.shell import client, vplan_templates

client.login(host=host, user=user, port=port, https=https)

vplan = vplan_templates.get_by_id(vplan_template_id=vplan_id)

from ftx.shell import client, vplan_templates
from ftx.model.search import Filter 

client.login(host=host, user=user, port=port, https=https)

filt = Filter.all("userName EQ example_user_name")
pagination = {"page": 0, "size": 200}

vplans = vplan_templates.page(filt=filt, pagination=pagination)

from ftx.shell import client, vplan_templates

client.login(host=host, user=user, port=port, https=https)

vplan_templates.delete(vplan_template_id=vplan_id)

74.4.7 Test Run Groups API

Provides methods for creating and analyzing a test run group (also known as Test Suite Result or TSR).

Examples:

from ftx.shell import client, test_run_groups

client.login(host=host, user=user, port=port, https=https)

group = test_run_groups.get_by_id(test_run_group_id=trg_id)

from ftx.shell import client, test_run_groups
from ftx.model.search import Filter

client.login(host=host, user=user, port=port, https=https)

filt = Filter.all("status EQ COMPLETED")
pagination = {"page": 0, "size": 200}

groups = test_run_groups.page(filt=filt, pagination=pagination)

from ftx.shell import client, test_run_groups
from ftx.model.search import Filter

client.login(host=host, user=user, port=port, https=https)

filt = Filter.all("status EQ STOPPED")

groups = test_run_groups.page(filt=filt)

from ftx.shell import client, test_run_groups
from ftx.model.search import Filter

client.login(host=host, user=user, port=port, https=https)

trg = test_run_groups.get_by_id(trg_id)
labels = trg['labels']
my_new_label = 'my_new_label'
new_labels = labels + [my_new_label] if isinstance(labels, list) else [my_new_label]

trg_after = test_run_groups.update(test_run_group_id=trg_id, labels=new_labels)

filt = Filter.all(f"labels CONTAINS_ANY {my_new_label}")

groups = test_run_groups.page(filt=filt)

74.4.8 Test Runs API

Provide methods for uploading runs to the Foretify Manager server and for analyzing them.

Examples:

from ftx.shell import client, test_runs

client.login(host=host, user=user, port=port, https=https)

run = test_runs.get_by_id(test_run_id=run_id, include_trace=True)

from ftx.shell import client, test_runs
from ftx.model.search import Filter

client.login(host=host, user=user, port=port, https=https)

filt = Filter.all("testRunGroupId EQ " + trg_id, "status EQ PASSED")
pagination = {"page": 0, "size": 200}

runs = test_runs.page(filt=filt, pagination=pagination, detailed=True, include_trace=True)

from ftx.shell import client, test_runs
from ftx.model.search import Filter

client.login(host=host, user=user, port=port, https=https)

test_run_group_id = trg_id
filt = Filter.all("testRunGroupId EQ " + test_run_group_id)
pagination = {"page": 0, "size": 200}

runs = test_runs.page(filt=filt, pagination=pagination, detailed=True)

issues = runs[['id', 'status', 'issues']]

from ftx.shell import client, test_runs

client.login(host=host, user=user, port=port, https=https)

runs = test_runs.get_by_workspace_test_run_filter(workspace_id=ws_id, detailed=True, include_trace=True)

from ftx.shell import client, test_runs
from ftx.model.search import Filter

client.login(host=host, user=user, port=port, https=https)

filt = Filter.any("testRunGroupId EQ " + trg_id)
runs = test_runs.get(filt=filt, detailed=True)

from ftx.shell import client, test_runs
from ftx.model.search import Filter, FilterContainer

client.login(host=host, user=user, port=port, https=https)

issue_1 = test_runs.Issue(
  category="some_category", 
  severity="WARNING",
  kind="some_kind", 
  details="some_details",
  time=1460,  # in milliseconds
  fullMessage="some_message",
  normalizedDetails="some_details", 
  modificationReason="some_reason",
  result="some_result"
)

issue_2 = test_runs.Issue(
  category="some_other_category", 
  severity="ERROR", 
  kind="some_other_kind", 
  details="some_other_details",
  time=600,  # in milliseconds 
  fullMessage="some_other_message", 
  normalizedDetails="some_other_details", 
  modificationReason="some_other_reason",
  result="some_other_result"
)

# add issues to a specific run by ID
test_runs.append_issues_by_id(test_run_id=run_id, issues=[issue_1, issue_2])

# add issues to multiple runs by regression ID and setting the main issue
test_runs.append_issues_by_filter(filter=FilterContainer.of(Filter.any("testRunGroupId EQ " + trg_id)), issues=[issue_1, issue_2], main_issue=issue_1)

from ftx.shell import client, test_runs

client.login(host=host, user=user, port=port, https=https)

runs = test_runs.get_by_id(run_id)
for interval in runs['intervals']:
    if interval['_type'] == 'CoverageIntervalData':
        interval_start_time = interval['startTime']
        interval_end_time = interval['endTime']
        interval_struct_and_group = interval['name']
        interval_scenarios = interval['ancestorScenarioNames']
        interval_items = interval['items']

from ftx.shell import client, test_runs

client.login(host=localhost, user=username, port=port)

runs = test_runs.get_by_id(run_id)
for interval in runs['intervals']:
    if interval['_type'] == 'WatcherIntervalData':
        interval_start_time = interval['startTime']
        interval_end_time = interval['endTime']
        interval_struct = interval['name']
        interval_scenarios = interval['ancestorScenarioNames']
        interval_watcher_name = interval['watcherName']
        interval_watcher_type = interval['watcherType']

74.4.9 Intervals API

Provide methods for fetching intervals from Foretify Manager server and for analyzing them.

74.5 Related Page

For more details, see Type relationship tables of compound intervals.

74.5.1 Common Parameters

• relation (in compound APIs):
  One of the following strings: "ANY_INTERSECTION", "A_BEFORE_B", "A_AFTER_B", "A_CONTAINS_B", "B_CONTAINS_A", "CUSTOM"
  

• temporal_action:
  An instance of the temporal_relations.TemporalAction enum class. Possible values are UNION or INTERSECTION.
  

• custom_relation:
  An instance of the temporal_relations.CustomTimeRelation class, which specifies constraints on the distances between two interval endpoints.
  For a full description, see temporal_relations.CustomTimeRelation in the temporal_relations module of ftx.shell.
  
  

Examples:

from argparse import ArgumentParser
from ftx._common.utils import add_common_arguments

def parse():
    parser = ArgumentParser(description="Getting arguments")
    add_common_arguments(parser)
    parser.add_argument("--regressions", nargs="+", help="regression ids")
    return parser.parse_args()

args = parse()
globals().update(vars(args))
trg_id = args.regressions[0]


from ftx.shell import client, intervals
from ftx.model.search import Filter, FilterContainer, RunFilter, IntervalFilter

client.login(host=host, user=user, port=port, https=https)

# fetch first 200 intervals under this tsr that are either watcher intervals or coverage intervals
filter_container = FilterContainer.of(
    RunFilter.of(Filter.all("testRunGroupId EQ " + trg_id)),
    IntervalFilter.of(filter=Filter.any("_type EQ WatcherIntervalData", "_type EQ CoverageIntervalData"))
)

results = intervals.page(filt=filter_container, pagination={"page": 0, "size": 200})

from argparse import ArgumentParser
from ftx._common.utils import add_common_arguments

def parse():
    parser = ArgumentParser(description="Getting arguments")
    add_common_arguments(parser)
    parser.add_argument("--regressions", nargs="+", help="regression ids")
    return parser.parse_args()

args = parse()
globals().update(vars(args))
trg_id = args.regressions[0]


from ftx.shell import client, intervals
from ftx.model.search import Filter, FilterContainer, RunFilter, IntervalFilter, ItemFilter

client.login(host=host, user=user, port=port, https=https)

test_runs_filter = FilterContainer.of(
    ItemFilter.of("sut.top.info", Filter.single("test EQ test")),
    RunFilter.of(filter=Filter.single(f'testRunGroupId EQ {trg_id}'))
)
interval_filter = FilterContainer.of(
    IntervalFilter.of(filter=Filter.single("_type EQ ScenarioIntervalData")),
)
# filter intervals in 2 steps. first filter runs with FilterContainer and then intervals by another FilterContainer
intervals.page(filt=interval_filter, test_run_filter=test_runs_filter)

from argparse import ArgumentParser
from ftx._common.utils import add_common_arguments

def parse():
    parser = ArgumentParser(description="Getting arguments")
    add_common_arguments(parser)
    parser.add_argument("--workspace", nargs="?", help="workspace id")
    return parser.parse_args()

args = parse()
globals().update(vars(args))
ws_id = args.workspace


from ftx.shell import client, intervals, workspaces
from ftx.model.search import Filter, IntervalFilter, ItemFilter, AdvancedIntervalFilter

client.login(host=host, user=user, port=port, https=https)

# add filter to workspace filters and fetch relevant intervals
interval_child_filter = AdvancedIntervalFilter.of(
    first_level_type=IntervalFilter.IntervalType.GlobalModifierIntervalData,
    children=[ItemFilter.of("top.info.main_issue", filter=Filter.single("result EQ Passed - no issue"))]
)
workspaces.filter_update(workspace_id=ws_id, children_relation_interval_filter=interval_child_filter)
matching_intervals = intervals.page(workspace_id=ws_id)

# resetting the filter
workspaces.filter_update(workspace_id=ws_id, children_relation_interval_filter=AdvancedIntervalFilter.empty())
matching_intervals = intervals.page(workspace_id=ws_id)

Creating a custom_relation which indicates "the distance between the starting point of the first interval to the starting point of the 2nd interval is between 1 and 2, and the distance between the endpoints is between 1 and 3":

from ftx.shell.temporal_relations import *

client.login(host=localhost, user=username, port=port)

custom_relation = temporal_relations.CustomTimeRelation(start_to_start=OpenRange(1,2), end_to_end=OpenRange(1,3))

74.5.2 Workspaces API

Provides methods for creating Foretify Manager Workspaces, changing their Runs Filter, and performing calculation tasks such as annotating them and ranking their corresponding runs.

Examples:

from ftx.shell import client, vplan_templates, workspaces

client.login(host=host, user=user, port=port, https=https)

vplan_base = vplan_templates.create(vplan_templates.collect("./demo/demo.vplan"))
workspace = workspaces.create(name="my_workspace_with_vplan_view",
                              vplan_template_id=vplan_base["id"],
                              project_id=project_id)
vplan_view_dict = {"name": "demo_vplan_view", "parentVplanTemplateId": vplan_base["id"], "source": "from_code"}
vplan_view = vplan_templates.create(vplan_view_dict)
workspace = workspaces.switch_vplan_view(workspace["id"], vplan_view["id"])  # Switch workspace to use the vplan view
workspace = workspaces.switch_vplan_view(workspace["id"], vplan_base["id"])  # Switch back to the base vplan template

74.5.2.1 Ranking

Notes:

• rank() waits for the ranking operation to finish. It returns a description of the ranking results (see below) and not the ranked runs themselves. The ranked test runs, paged and ordered by their cumulative grade (rank), are returned when calling ranked_runs().

• rank_limit and target_grade are the ranking calculation's limits: it will stop evaluating additional runs as soon as one of these parameters is achieved (i.e. either evaluated rank_limit runs or the cumulative grade of the run just evaluated achieved target_grade).

• vplan_path is a dictionary containing the fields sections and element, both are lists. For example, to rank the top.section1.section2 section:

  vplan_path = {
      "sections" :["top", "section1","section2"],
      "elements": []
  }
  

  Or, to rank the cut_in.end.speed item under the top.section1.section2 section:

  vplan_path = {
      "sections" :["top", "sections1", "section2"],
      "elements": ["cut_in", "end", "speed"]
  }
  

• grade_type is the grading scheme to be used while ranking (when calculating the cumulative grade). The possible types are: 1) AVG: The averaging grade scheme 2) BUCKET: The bucket grade scheme 3) PROG_BUCKET: The progressive bucket grade scheme For detailed information about the schemes, see Workspace grading object.

• The results of a rank() request is an object containing a description of the requested rank (rankedNodeName, rankLimit & targetGrade) and the results of the rank (cumulativeGrade & rankedRuns), for example:

  rankings = {
      "rankedNodeName": "top.section1.section2.cut_in.end.speed",
      "rankLimit": 1000,
      "targetGrade": 100.0,
      "cumulativeGrade": 88.53,
      "rankedRuns": 1000
  }