76. Foretify Manager utilities

Foretify Manager has various utilities the <fmanager-install-path>/client/bin directory.:

Utility	Description	User role
fmanager_shell	Enable opening Foretify from Foretify Manager. See Foretify Manager API for details.	USER
download_regression	Download a regression from Foretify to a local directory. See download_regression.	USER
list_users	Obtain information about logged in users and licenses consumption. See list_users.	USER
upload_runs	Upload a regression run by FRun to Foretify Manager. See upload_runs.	DATA_UPLOADER
upload_vplan	Upload a VPlan to Foretify Manager. See upload_vplan.	DATA_UPLOADER
create_user	Create user accounts. See create_user.	ADMIN
change_password	Change a user's password. See change_password.	USER
launch_predefined_regression	Start a regression using Dispatcher by an existing test plan. See launch_predefined_regression.	DATA_UPLOADER
launch_regression	Start a regression using Dispatcher by the provided environment settings and FRun files. See launch_regression.	DATA_UPLOADER
upload_concrete_tests	(Early Access) Upload concrete scenario YAML test files to Foretify Manager. See upload_concrete_tests.	USER

Depending on the privileges for your user account, you might not be able to use all these utilities:

ADMIN: Full administrative privileges including creating user accounts, data upload to the server and data analysis. A license is checked out at login or use of python utilities.
USER: Full user privileges in including data upload to the server and data analysis. A license is checked out at login or use of python utilities. This is the default.
DATA_UPLOADER: Allowed only to upload VPlans or run data to the server. No license is checked out.

In any case, in order to use these utilities, set the following environment variables:

Shell script: set FMANAGER env vars

export FTX_FMANAGER_HOST="<ip-address>"
export FTX_FMANAGER_PORT="<port>"
export FMANAGER="<fmanager-install-path>"
export PATH="$PATH:$FMANAGER/client/bin"

See the following sections for descriptions of the utilities.

76.1 download_regression

Purpose

Download a regression (also known as Test Suite Results) from Foretify to a local directory. The user role needed to download a regression is USER.

Note

The download_regression utility is a support tool for uploading user data. As it is sensitive to network slowness and stability and is heavy on CPU usage, it should be used only as a support tool.

Syntax

download_regression [--help] --id ID [ID ...] --dest DEST
                    [--extra_data] [--file_filter FILE_FILTER]
                    [--host HOST] --port PORT [--https] [--user USER]

Arguments

--id ID [ID ...]: (Required) Regression Test Suite results (TSR) ID.
--dest DEST: (Required) Target directory for downloaded file.
--extra_data: (Optional) Fetch extra (debug) data of runs, if available.
--file_filter FILE_FILTER: (Optional) Include only filenames matching the specified wildcard pattern(s). Wildcard patterns must be comma-separated. To use --file_filter, you must also set --extra_data.
--host HOST: (Optional) Foretify Manager host name. Default host is localhost.
--port PORT: (Required) Foretify Manager port number. Specify port 8080 for protocol http and port 443 for protocol https.
--https: (Optional) Foretify Manager https protocol. If not specified, https is used.
--user USER: (Optional) Username.

Example

Foretify Manager utility: download_regression

$ download_regression --id 4cb7b1000132a04c8219 --dest ~/working \
                    --host 169.23.41.523 --port 8080 \ 
                    --user "mylogin"
SDK version: 23.10.0.17
Connecting to Foretify Manager @ http://169.23.41.523:8080 with user 'mylogin'...
... logged in (server is UP, version: develop.db8146aa79e)
Downloading...: 80.0kiB [00:00, 4.30MiB/s]
Successfully downloaded '/users/mylogin/working/kareng_run_group_07-02-2023-131006-2023-10-25_18-22-40.zip'

76.2 list_users

Purpose

Obtain information about logged in users and licenses consumption. The user role needed to list users is USER.

Syntax

list_users [--help] [--host HOST] --port PORT [--https] [--user USER]

Arguments

--host HOST: (Optional) Foretify Manager host name.
--host PORT: (Required) Foretify Manager port number. Specify 8080 for protocol http and 443 for protocol https.
--https: (Optional) Foretify Manager https protocol. If not specified, https is used.
--user USER: (Optional) Username.

Example

Foretify Manager utility: list_users

$ list_users --host 169.23.41.523 --port 8080 \
           --user "mylogin"
Connecting to Foretify Manager @ http://169.23.41.523:8080 with user 'mylogin'...
... logged in (server is UP, version: develop.db8146aa79e)
User: jennas, No. of logins: 1 (1 checked out licenses)
User: jamesf, No. of logins: 2 (2 checked out licenses)

76.3 upload_runs

Purpose

Upload a regression run by FRUN to Foretify Manager. The user role needed to upload runs is DATA_UPLOADER.

Syntax

upload_runs [--help] [--host HOST] --port PORT [--https] [--user USER]
            [--run_group_name RUN_GROUP_NAME]
            [--runs_top_dir RUNS_TOP_DIR [RUNS_TOP_DIR ...]]
            [--copy_top_dir_to COPY_TOP_DIR_TO [COPY_TOP_DIR_TO ...]]
            [--preserve_paths]
            [--to_s3_url S3_URL]
            [--from_s3_url S3_URL]
            [--azure_storage_url AZURE_STORAGE_URL]
            [--project_id PROJECT_ID]
            [--project [PROJECT [PROJECT ...]]]
            [--labels [LABELS [LABELS ...]]]
            [--interactive]

Arguments

--host HOST: (Optional) Foretify Manager host name. Default host is localhost.
--port PORT: (Required) Foretify Manager port number. Specify port 8080 for protocol http and port 443 for protocol https.
--https: (Optional) Foretify Manager https protocol. If not specified, https is used.
--user USER: (Optional) Username.
--run_group_name RUN_GROUP_NAME: (Optional) Regression (Test Suite Results) name containing the uploaded runs.
--runs_top_dir RUNS_TOP_DIR [RUNS_TOP_DIR ...]: (Optional) Top directory containing all runtime sampling protobuf files.
--copy_top_dir_to COPY_TOP_DIR_TO [COPY_TOP_DIR_TO ...]: (Optional) Copy the top runs directory specified by the runs_top_dir argument to a specified path.
--preserve_paths: (Optional) Preserve paths in original run info.
--from_s3_url S3_URL: (Optional) AWS S3 URL (bucket and prefix) to retrieve runs from.
--to_s3_url S3_URL: (Optional) AWS S3 URL (bucket and prefix) to upload runs to.
--azure_storage_url AZURE_STORAGE_URL: (Optional) Azure blob storage URL to retrieve runs from.
--project_id PROJECT_ID: (Optional) Target project ID of the regression.
--project [PROJECT [PROJECT ...]]: (Optional) Target project name of the regression.

--labels [LABELS [LABELS ...]]: (Optional) Labels for the regression.
--skip_schema_version_validation: (Optional) When present, Foretify Manager allows uploading of test runs that do not fit the current schema version (i.e, outdated runs).
--interactive: (Optional) When provided, uploads from S3 will display a progress bar indicating the upload's progress. Otherwise, the upload to S3 will be non-blocking and will return immediately with a task ID, which can be used later to track the upload progress using the SDK function test_runs.get_s3_upload_progress(task_id).

upload_runs argument rules:

Specify at least one of the following arguments:
- runs_top_dir
- model
- from_s3_url
- azure_storage_url
Specify at least one of the following arguments:
- project_id
- project
- model
If you specify copy_top_dir_to, you must also specify runs_top_dir.
The following arguments cannot be used together:
- runs_top_dir and from_s3_url
- runs_top_dir and azure_storage_url
- from_s3_url and azure_storage_url
- project_id and project
Specify only one of the following S3-related arguments:
- s3_bucket
- to_s3_url
- from_s3_url

Example

Foretify Manager utility: upload_runs

$ upload_runs --user "mylogin" \
  --host 169.23.41.523 --port 8080 \
  --runs_top_dir myworkdir/sessions/20221123_150125_10395_4B60/20221123_150125_25680_WD19/foretify_runs/batch \
  --project "My Team's Project"

76.4 upload_vplan

Purpose

Upload a VPlan to Foretify Manager. The user role needed to upload a VPlan is DATA_UPLOADER.

Syntax

upload_vplan [--help] --vplan_template VPLAN_TEMPLATE [VPLAN_TEMPLATE ...]
             [--host HOST] --port PORT [--https]
             [--user USER]

Arguments

--vplan_template VPLAN_TEMPLATE [VPLAN_TEMPLATE ...]: (Required) VPlan template JSON file.
--host HOST: (Optional) Foretify Manager host name. Default host is localhost.
--host PORT: (Required) Foretify Manager port number. Specify port 8080 for protocol http and port 443 for protocol https.
--https: (Optional) Foretify Manager https protocol. If not specified, https is used.
--user USER: (Optional) Username.

Example

Foretify Manager utility: upload_vplan

$ upload_vplan --vplan_template ./hw_lib.json \
--host 169.23.41.523 --port 8080 \
--user "mylogin" \
SDK version: 23.10.0.17
Connecting to Foretify Manager @ http://169.23.41.523:8080 with user 'mylogin'...
... logged in (server is UP, version: develop.db8146aa79e)
uploaded 1 vplan templates

You can also download a VPlan as a JSON file, update it, and upload it.

76.5 create_user

Purpose

Create user accounts. The user role needed to create user accounts is ADMIN.

Syntax

create_user [--help] [--host HOST] --port PORT [--https] --user USER
            [--display_name DISPLAY_NAME] [--role ROLE] [--email EMAIL]
            [--admin_user ADMIN_USER] [--admin_access_token ADMIN_ACCESS_TOKEN]

Arguments

--host HOST: (Optional) Foretify Manager host name. Default host is localhost.
--host PORT: (Required) Foretify Manager port number. Specify 8080 for protocol http and 443 for protocol https.
--https: (Optional) Foretify Manager https protocol. If not specified, https is used.
--user USER: (Required) Username.
--display_name DISPLAY_NAME: (Optional) Display name.
--role ROLE: (Optional) Role.
--email EMAIL: (Optional) Email address.
--admin_user ADMIN_USER: (Optional) Administrator User's username.
--admin_access_token ADMIN_ACCESS_TOKEN: (Optional) Administrator User's access token.

Example

Foretify Manager utility: create_user

$ create_user \
--host 169.23.41.523 --port 8080 \
--user jamesf \
--email jamesf@acme.com \
--admin_user my_admin_user \

Note: Administrator credentials can be given either by username and password, access token, or .ftxpass file. When using .ftxpass, no credentials should be given in the bash command.

76.6 change_password

Purpose

Change a user's password. The user role needed to change a user's password is USER.

Syntax

change_password [--help] [--host HOST] --port PORT [--https]
                --user USER

Arguments

--host HOST: (Optional) Foretify Manager host name. Default host is localhost.
--host PORT: (Required) Foretify Manager port number. Specify 8080 for protocol http and 443 for protocol https.
--https: (Optional) Foretify Manager https protocol. If not specified, https is used.
--user USER: (Required) Username.

Example

Foretify Manager utility: change_password

$ change_password --user jamesf --host 169.23.41.523 --port 8080

76.7 launch_predefined_regression

Purpose

Launch a test suite by starting a regression (also known as Test Suite Results) using Dispatcher with an existing test plan. The user role needed to start a regression is USER.

Syntax

launch_predefined_regression [--help] [--host HOST] --port PORT
                             [--https] [--user USER]
                             [--name NAME] [--labels [LABELS [LABELS ...]]]
                             --definition_name DEFINITION_NAME
                             [--environment_variables [ENVIRONMENT_VARIABLES [ENVIRONMENT_VARIABLES ...]]]
                             [--project_id PROJECT_ID]
                             [--project [PROJECT [PROJECT ...]]]
                             [--job_labels [JOB_LABELS [JOB_LABELS ...]]]

Arguments

--host HOST: (Optional) Foretify Manager host name. Default host is localhost.
--host PORT: (Required) Foretify Manager port number. Specify 8080 for protocol http and 443 for protocol https.
--https: (Optional) Foretify Manager https protocol. If not specified, https is used.
--user USER: (Optional) Username.
--name NAME: (Optional) Regression name (automatically generated if not given).
--labels [LABELS [LABELS ...]]: (Optional) Labels to apply on the regression.
--definition_name DEFINITION_NAME: (Required) An existing test plan name to launch.
--environment_variables [ENVIRONMENT_VARIABLES [ENVIRONMENT_VARIABLES ...]]: (Optional) Environment variables to be appended to the environment settings. Each variable should be given as KEY=value.
--project_id PROJECT_ID: (Optional) Target project ID of the regression.
--project [PROJECT [PROJECT ...]]: (Optional) Target project name of the regression.
--job_labels [JOB_LABELS [JOB_LABELS ...]]: (Optional) Job labels to be appended to the environment settings. Each label should be given as KEY=value.

In the above command:

Specify only one of the following arguments: project_id or project.

Example

Foretify Manager utility: launch_predefined_regression

$ ./launch_predefined_regression --host localhost --user my_user \
    --name my_launched_regression \
    --definition_name my_def \
    --environment_variables FTX_PACKAGES=/ftx/packages \
    --project "My Team's Project" \
    --job_labels job_label_1=value job_label_2=value2 \ 
    --labels test_suite_label_1 test_suite_label_2

76.8 launch_regression

Purpose

Launch a new regression (also know as Test Suite Results) with a newly-generated definition by starting a regression using Dispatcher with the provided environment settings and FRun files. The user role needed to start a regression is USER.

Syntax

launch_regression [--help] [--host HOST] --port PORT [--https]
                  [--user USER] [--name NAME]
                  [--labels [LABELS [LABELS ...]]]
                  --files FILES [FILES ...]
                  [--environment_settings_name ENVIRONMENT_SETTINGS_NAME]
                  [--environment_variables [ENVIRONMENT_VARIABLES [ENVIRONMENT_VARIABLES ...]]]
                  [--job_labels [JOB_LABELS [JOB_LABELS ...]]]
                  [--preserve_definition] [--project_id PROJECT_ID]
                  [--project [PROJECT [PROJECT ...]]]
                  [--environment_settings_path ENVIRONMENT_SETTINGS_PATH]

Arguments

--host HOST: (Optional) Foretify Manager host name. Default host is localhost.
--host PORT: (Required) Foretify Manager port number. Specify 8080 for protocol http and 443 for protocol https.
--https: (Optional) Foretify Manager https protocol. If not specified, https is used.
--user USER: (Optional) Username.
--name NAME: (Optional) Regression name (automatically generated if not given).
--labels [LABELS [LABELS ...]]: (Optional) Labels to apply on the regression.
--files FILES [FILES ...]: (Required) One or more FRun files (either csv or txt).
--environment_settings_name ENVIRONMENT_SETTINGS_NAME: (Optional) Name of existing environment settings.
--environment_variables [ENVIRONMENT_VARIABLES [ENVIRONMENT_VARIABLES ...]]: (Optional) Environment variables to be appended to the ones already defined in environment settings. Each variable should be given as KEY=value and multiple variables should be separated by spaces.
--job_labels [JOB_LABELS [JOB_LABELS ...]]: (Optional) Job labels to be added to Dispatcher jobs created for this test suite. Each label should be given as KEY=value.
--preserve_definition: (Optional) If specified, test run group definition will not be deleted.
--project_id PROJECT_ID: (Optional) Target project ID of the regression.
--project [PROJECT [PROJECT ...]]: (Optional) Target project name of the regression.
--environment_settings_path ENVIRONMENT_SETTINGS_PATH: (Optional) Path to a JSON file representing the environment settings.

In the above command:

Specify at least one of the following arguments: environment_settings_name or environment_settings_path.
Specify only one of the following arguments: project_id or project.

Example

Foretify Manager utility: launch_regression

$ ./launch_regression --host localhost --user my_user \
    --name my_launched_regression \
    --files /my/shared/folder/regression.csv \
    --environment_settings_name sumo:22.12.0 \
    --environment_variables a=v c=d \
    --project "My Team's Project" \
    --job_labels job_label_1=value job_label_2=value2 \
    --labels test_suite_label_1 test_suite_label_2

76.9 upload_concrete_tests

This utility is currently in early access and may be subject to changes in future releases.

Purpose

Upload concrete scenario YAML test files to Foretify Manager. This utility scans a directory recursively for YAML files and uploads them as tests with type CONCRETE. The user role needed to upload concrete tests is USER.

Syntax

upload_concrete_tests [--help] [--host HOST] --port PORT [--https]
                      [--user USER] [--access_token ACCESS_TOKEN]
                      --tests_top_dir TESTS_TOP_DIR

Arguments

--tests_top_dir TESTS_TOP_DIR: (Required) Top directory to scan recursively for concrete scenario YAML files (.yaml or .yml extensions).
--host HOST: (Optional) Foretify Manager host name. Default host is localhost.
--port PORT: (Required) Foretify Manager port number. Specify port 8080 for protocol http and port 443 for protocol https.
--https: (Optional) Foretify Manager https protocol. If not specified, http is used.
--user USER: (Optional) Username.
--access_token ACCESS_TOKEN: (Optional) Access token for authentication (alternative to username/password).

Example

Foretify Manager utility: upload_concrete_tests

$ upload_concrete_tests --host 169.23.41.523 --port 8080 \
    --user "mylogin" \
    --tests_top_dir /path/to/concrete/scenarios