75. Python Rules for Triage

The Foretify Manager Triage UI provides capabilities for creating and assigning attributes for user assignment, bug tracking, setting triage verdicts, and more. You can use the Foretify Manager UI to set rules like the following:

Always assign issues that match a condition to a specific user.
Associate issues that match a condition to a link from your bug tracking system.
Set custom attributes based on other attributes (for example, 'issue sub-kind').

See Work with Triage rules to learn how to set these types of rules and more.

However, there are some advanced rules that can't be created and applied solely within the Foretify Manager UI. In such cases where the expressive power of the Foretify Manager UI rules is inadequate, you can create and apply Python Rules. With Python Rules, for example, you can assign an attribute's value based on:

Other attributes

Example: A run is considered too short (too_short=yes) if simulation_time <= 3.5.
Metric items saved by the .osc test script

Example: The time when the SUT reaches an autonomous state is extracted from metric group top.info.end item time_ego_reached_autonomous_state.
Comparisons of a run with the corresponding run

Example: If the difference between simulation times of this and the compared runs is more than 10%, the runs probably do not represent the same behavior (even if the result is the same).

When using Python Rules, you write a python script (rules.py) contains the logic to assign values to the run's attributes. If your Python logic is split between several files, the rules.py file can import other Python files from the same directory. Your python script (rules.py) and other supporting files, should be placed in one directory which is referred as ‘tdoc’ directory. You can define new attributes in the script, and these are later available when you create custom Triage views in Foretify Manager.

To apply the python rules, upload them as the “Active python triage rule” of the workspace, and then click Apply. For more information, see Python rules and Applying Rules. If your Fmanager server is not configured with Python rules, you can run Python rules from command line, for more information, see Applying python rules through command line.

After the Python Rules utility completes, you can see the results in the Foretify Manager Triage view. You may also want to create custom views that show the updated attributes. By creating custom Foretify Manager views, you can use the new attribute values to group runs in new ways.

75.1 Rules file

The rules file is a Python file, named rules.py. The file must be in a directory that will be called "the tdoc directory" in this document. If you need different rules files for different scenarios, put each in its own directory. (It makes sense to name each directory according to the scenario the rules file covers.) Also, if your Python logic is split between several files, the rules.py file can import other Python files from the same directory.

The rules.py file contains Python methods with predefined names that are activated by the system once for each run in the “Triaged test suite results.” These methods can set values for the run’s attributes.

preliminary_defs(r): This method (if it exists in rules.py file) is activated once for each run during the “Apply preliminary Python triage rules” phase.

basic_defs(r), extra_defs(r), final_defs(r): These methods (if they exist in rules.py file) are activated once for each run during the “Apply other Python triage rules (basic, extra, final)” phase, in this order: basic_defs(r) → extra_defs(r) → final_defs(r). Generally, it is not necessary to provide all three methods; most logic can be implemented using just basic_defs(r).

Following is a sample rules.py file (located in python_rules/tdocs/sample_tdoc).

Python code: sample rules.py file
import tman as tman
# This import must always exist

# ----- This is an example of rules file showing the different features of Python Rules
def basic_defs(r):
   """These are executed for each run.
   @param r: Represents a run. All attributes are fields of this run.
             Attribute names are snake_case, not camelCase (e.g., r.test_name, not r.testName)
   """
   tman.var("is_long_test_name", "Is Name Long?")
   # This attribute holds yes/no, depending on the length of the test name
   if len(r.test_name) > 20:
       r.is_long_test_name = "yes"
   else:
       r.is_long_test_name = "no"

   tman.var("cut_in_side", "Cut In Side")
   # This attribute holds the side of the cut-in (left or right)
   # It is an example of how to use metric group items
   # r.cut_in_side will contain None if the metric group item does not exist
   r.cut_in_side = tman.get_item(r, "sut.cut_in.end", "side")

   tman.var("time_diff", "Time Diff")
   # This attribute will hold indication '(not same time)' if the time of the current run
   # is significantly different from the time of the previous run
   # It will be empty otherwise.
   # This is an example of comparing to the previous run
   prev = tman.cr(r)  # This returns the Corresponding Run (CR) from the Compared TSR (if exists)
   if prev is not None:
       if r.simulation_time > 0 and prev.simulation_time > 0:
           time_diff = abs(r.simulation_time - prev.simulation_time) / r.simulation_time
           if time_diff > 0.1:
               r.time_diff = '(not same time)'
   else:
       r.time_diff = '(no previous run'

Following is a description of the sample rules.py file:

Line 1: import tman as t

This line must always exist. The infrastructure makes the module, tman, available when you run python-rules.
Line 4: def basic_defs(r)

This method is called for each run in the Test Suite Results (TSR). r is the run object.
Line 9: tman.var("is_long_test_name", "Is Name Long?")

This is an example of defining a custom attribute. Custom attributes must be defined before being used. If the attribute exists in Foretify Manager, the existing attribute is used, otherwise the attribute is automatically created in Foretify Manager.
- The first parameter is the attribute's internal name which is used to set and read values of the new attribute in rules.py.
- The second parameter is the Display Name which is shown in the Foretify Manager UI.
- A named description parameter is also supported. It sets the attribute description shown in the Foretify Manager UI, for example:
```
tman.var("is_long_test_name", "Is Name Long?", 
       description="Identifies runs with a long name")
```
Line 11: if len(r.test_name) > 20

This is an example of using the value of a built-in (predefined) attribute (r.test_name). To see a complete list of the predefined triage attributes, see Attributes.
Line 12: r.is_long_test_name = "yes"

This is an example of setting the value of an attribute.
Line 20: tman.get_item(r, "sut.cut_in.end", "side")

The tman.get_item(...) method is the way to access metric group items saved by the .osc test script. The parameters are:
- r: The run object
- "sut.cut_in.end": The group name
- "side" - The metric item
If there are multiple metric group items with the same group and item name, the value of the last (most recent) one is returned. If there is no matching metric group item, Python None is returned.
Line 27: tman.cr(r)

tman.cr(r) returns a run object that is the corresponding run from the compared TSR. You can access values of system or custom attributes and even access metric items of this run, but you cannot set any attribute values.

The system automatically finds the corresponding run from among the runs in the compared TSR. (The compared TSR is set in the Triage view of the Foretify Manager UI. See Compare test suite results.) A run is considered the corresponding run if the test_name and seed matches those of r (the current run).

If no corresponding run exists in the compared TSR, or if no compared TSR is set, tman.cr(r) returns Python None.
tman.get_item_list(r, group_name, item_name)

The tman.get_item_list(...) method (which doesn't appear in the sample rules) is the way to access repeated metric group items saved by the .osc test script. The parameters are the same as for tman.get_item(...) above, but the return value is always a Python list:
- If there are multiple metric group items with the same group and item name, the values are returned in a Python list.
- If there is no matching metric group item, an empty Python list is returned.

75.2 Applying python rules through command line

If your Fmanager server is not configured with Python rules, you can run Python rules from the command line. See below for instructions on how to install and use the command-line utility.

75.2.1 Install Python Rules

The latest version of Python Rules is located in the /data/tools/fmanager/latest/client/bin/ directory.

You must install Python Rules on your computer with access to the fmanager (Foretify Manager) that the rules will be applied to.

Following are the system requirements for python_rules:

Ubuntu 22.04
Python 3.8 or 3.9

To install Python Rules:

Copy python_rules from /data/tools/fmanager/latest/client/bin/ to a directory on your computer. Note that this is a large (~80MB) file and the copy may take some time.

75.2.2 python-rules utility

Purpose

To use Python Rules, you run the python-rules utility which lets you assign values to a run's attributes, as well as create new attributes that you can use in custom Triage views.

Syntax

Note

Ensure that the current directory, containing the python-rules utility, is in your execution path, or use ./python-run to reference it explicitly.
When applying triage rules on a Windows machine, the system cannot open a JSZip file if its name or path is longer than 260 characters.

./python-rules -t [tdoc directory] -w [workspace ID] -f [Foretify Manager host/IP] -tp [TSR ID] -d -u [Foretify Manager username] -p [Foretify Manager password] -D

Required parameters

-t [tdoc directory]: Full or relative path to a local directory that contains the rules.py file. See Rules file for details.
-w [workspace ID]: ID of the workspace to act upon. The workspace ID can be taken from the URL when in the Foretify Manager Triage view, for example: http://111.11.11.11:8080/workspaces/82c2be07c0d6383685e2/triageManager. The workspace ID is the 16-digit identifier between workspaces/ and /triageManager in the URL. You can also see the workspace ID in the Foretify Manager project view (which lists the workspaces in your project) by adding the ID column to the displayed columns. See how to add columns.
-f [Foretify Manager host/IP]: IP address or host name of your Foretify Manager installation (without the :8080 port specifier). Here is how it appears in the URL: http://111.11.11.11:8080/workspaces/45c2be07c0d6383685e2/triageManager (this is what appears between http:// and :8080).

Optional parameters

-tp [TSR ID]: Sets the timeline points to act upon. A timeline point represents a single TSR or multiple TSRs that were grouped. The default is the timeline point that the workspace is using as the triaged TSR. You can see the timeline point ID in the Foretify Manager workspace by clicking the Workspace Test Suite Results icon on the left sidebar and adding the timeline point ID column to the displayed columns. See how to add columns. You can also group several TSRs into one timeline point on that page. (For use with Python Rules, only group TSRs that have different test names in their runs.)
-d: Remove duplicate runs. If several runs in the TSR have the same <test name, seed> pairs, only one of them will be used. Removing duplicates is recommended when you use rules that compare with the previous TSR — otherwise, there is no one-to-one matching of runs between the previous and current TSRs. With this option turned on, if any runs are ignored (because they are duplicates), Python Rules reports the number of ignored runs.
-u [Foretify Manager username]: You can specify your Foretify Manager username on the command line; otherwise, Python Rules prompts for your username.
-p [Foretify Manager password]: You can specify your Foretify Manager password on the command line; otherwise, Python Rules prompts for your password.
-D: Delete attributes created in Foretify Manager during the last Python Rules run with the same tdoc. This is useful during development of the rules.py rules file when you do not wish to have many attributes that are not yet useful. Also note that this will only delete attributes created during the last regular run (the run without -D). Also note that if -D is specified, no rules are calculated and no attribute values are updated—the only thing that happens is that custom attributes are deleted from Foretify Manager. When using -D, the parameter -w [workspace ID] is ignored.

Example:

`./python-rules -t tdocs/sample_tdoc/ -w 4325bc73267383f276a0 -f 111.11.11.11`

After Python Rules activation completes, the Foretify Manager Triage rules (those that are set through the Foretify Manager UI) are automatically applied.

This means that if the Python Rules and the Foretify Manager Triage rules set contradicting values, the Foretify Manager Triage rules "win".
This also means that if attribute values that are used in a Foretify Manager Triage rule filter are modified by Python Rules, reapplying the Foretify Manager Triage rules takes these new values into account.