Skip to content

89. ESR Kit integration with the Foretify Manager Dispatcher

The Evaluation Solution Reference (ESR) Kit provides a a python-based integration with the Foretify Manager Dispatcher, to enable large-scale parallel execution and workflow orchestration of the evaluation pipeline stages. This integration is referred to as the ESR Kit Dispatcher Integration.

89.1 ESR Kit Dispatcher Integration Overview

The ESR Kit Dispatcher Integration is engineered to streamline large-scale evaluation workflows by combining robust batch processing capabilities with a flexible and staged architecture. Here are the highlights of the integration:

  • Parallel batch processing

    The ESR Kit Dispatcher Integration executes multiple evaluation jobs simultaneously for efficient throughput, allowing users to process high volumes of data with minimum manual intervention.

  • Staged workflow

    Evaluation is structured into discrete stages, such as parse, ingest, and match, each clearly defined within user-editable YAML configuration files. The tool advances from one stage to the next only after evaluating the outcomes of all jobs in the preceding stage, ensuring higher reliability and fault tolerance.

  • Dispatcher integration and job management

    Leveraging the Dispatcher framework, the ESR Kit Dispatcher Integration creates, schedules, and monitors jobs for each input. Job statuses are tracked closely, and only validated outputs progress through subsequent workflow stages.

  • Configurable and scalable architecture

    Jobs and batch parameters are defined in configuration files, supporting customization for a wide range of workflows. The architecture is optimized for scalability: During the ingestion phase, the tool automatically iterates through an input directory and assigns dispatcher jobs for each file.

  • Results evaluation and advancement

    After completion of jobs in each stage, the tool evaluates the results against predefined success criteria. This evaluation ensures that only successful and complete job outputs lead to the initiation of the next processing stage.

89.2 Prerequisites for using the ESR Kit Dispatcher integration

Before you can use the ESR Kit Dispatcher Integration, make sure the following prerequisites are met:

  1. [Set up the ESR Kit environment][setting-up-the-esr-reference-kit-environment].

    This is a required prerequisite to ensure that all necessary data paths and configurations are correctly initialized.

    Warning

    Skipping or misconfiguring the environment setup will result in failures when running batch evaluations.

  2. Prepare the required YAML configuration files.

    The ESR Kit Dispatcher Integration requires two YAML configuration files.

    • The ref_kit_config.yaml file defines core evaluation jobs for the ESR Kit. See Configuring the ref_kit_config.yaml file.

    • The batch_config.yaml file is described in detail below. This file contains batch-specific settings.

89.3 Configuring the Batch YAML file

The batch_config.yaml file must be configured for each run. It contains the settings used to orchestrate multiple jobs at scale.

Note

Ensure your ref_kit_config.yaml is already configured with your evaluation parameters before proceeding. See Configure the ref_kit_config.yaml file.

  1. Copy the my_batch_config.yaml template to your working directory:

    cp $FTX/ref_kit/batch/configs/batch_eval_config.yaml ./my_batch_config.yaml

  2. Open my_batch_config.yaml in your editor, and configure the parameters.

    Refer to the full ESR Kit Dispatcher Integration YAML configuration for all configurable parameters.

  3. Double-check permissions. Verify on both your host and the execution environment that all directories are readable and writable by the batch tool and any spawned jobs.

  4. The ESR Kit Dispatcher Integration is run using the evaluate command.

To start a new batch evaluation:

$FTX/bin/evaluate/evaluate --config refkit_config.yaml --batch batch_config.yaml

To resume an interrupted run using group ID:

$FTX/bin/evaluate/evaluate --group-id <GROUP_ID>

89.4 ESR Kit Dispatcher Integration YAML configuration

Below is a complete reference of all parameters available in the ESR Kit Dispatcher Integration configuration YAML file.

Parameter Description
dispatcher_url API endpoint for of the dispatcher (e.g., http://<hostname-or-ip>:8081).
docker_image_tag Specifies the Foretify Docker image tag to use for all jobs.
ftx_lic_file The Foretify license server information (e.g., 30001@<hostname-or-ip>)
ftx_esr_kit_home Path to the ESR installation location within the job container (e.g., /ftx/ref_kit)
match_results_dir Optional output directory to copy all final match (evaluation) results and artifacts to.

89.5 Best Practices for the ESR Kit Dispatcher Integration

  • Always review and validate your YAML configuration files before starting a batch run to ensure correct job definitions.
  • Paths must be accessible and writable for job plugins.
  • Monitor job logs throughout batch evaluation to quickly identify and troubleshoot issues at each processing stage.
  • Take advantage of grouping and resume features for managing complex evaluation workflows and re-running failed or interrupted jobs.