517. Build a docker image for a complete Foretify Manager environment

Foretify Manager provides a Dockerfile that lets you build a docker image containing all components required for running Foretify Manager, including dispatching capabilities.

To build a complete dockerized environment, do the following:

1. Build a Foretify Manager docker image.

2. Run the Foretify Manager docker image.

3. Download and run the Elasticsearch and Postgres docker images.

4. Run the above three images using Kubernetes orchestration.

517.1 Build a Foretify Manager docker image

Prerequisites:

• A docker executable is available.

• The docker executable’s directory is included in $PATH.

• Depending on your user environment you may need superuser permissions to run Docker. In that case, run the docker commands shown below with sudo.

The Foretify Manager installation contains a docker directory in the installation directory: <install path>/docker. This directory contains the following files:

To build a Foretify Manager image, use the following command:

$ <install path>/docker/create_image.sh [-t | --tag <tag_name>]

$ /opt/foretellix/fmanager/docker/create_image.sh fmanager-image:ver1

The <tag_name> is an optional argument that can be used to give a tag to the created image; otherwise, a name is given by docker.

To see all options of the create_image.sh script, use the -h or --help option.

517.2 Run Foretify Manager docker image

To run the Foretify Manager docker image, use the docker run command. As part of this command invocation:

• Some physical volumes should be mounted to the docker container with the -v option.

• Environment variables should be set with the -e option.

Here is an example of how to use the docker run command to run the Foretify Manager docker image:

$ docker run -v /my_var/log/foretellix:/var/log/foretellix \
-v /my_opt/foretellix/fmanager/config:/opt/foretellix/fmanager/config \
-v /my_opt/foretellix/dispatcher/config:/opt/foretellix/dispatcher/config \
-v /my_opt/foretellix/dispatcher/jobs:/opt/foretellix/dispatcher/jobs \
-v /my_opt/foretellix/dispatcher/shared:/opt/foretellix/dispatcher/shared \
--network="host" \
-e FTX_LIC_FILE="30001@172.31.49.28" fmanager:ver1 8080:8080

The -v option is used for mounting physical volumes to volumes that will be used as part of the docker container runtime environment. For example, the command below mounts /my_var/log/foretellix to the container environment. From within the container, this volume is referred to as /var/log/fortellix.:

  -v /my_var/log/foretellix:/var/log/foretellix

The -e option is used for passing environment variables to the container environment. You must set a license server for Foretify Manager using the following syntax:

    -e FTX_LIC_FILE="<port>@<ip>"

The –network=”host” option is not mandatory, but it ensures that the container uses the host machine network. For example, “localhost” from within the container context points to the host machine.

In the example above fmanager:ver1 is the docker image name, and 8080:8080 ensures that port 8080, which is the default port for accessing Foretify Manager, is externalized and can be used outside of the container environment.

517.3 Run Elasticsearch and Postgres docker images

Foretify Manager relies on user installations of two databases:

• Postgres

• Elasticsearch

The following sections provide basic examples of how to run Elasticsearch and the Postgres database in a container.

517.3.1 Run a Postgres docker image

Docker Hub provides a full catalog of Postgres images (https://hub.docker.com/_/postgres) that you can pull into a user local docker registry.

Instructions for configuring the Postgres image when running it in a docker container are defined in https://hub.docker.com/_/postgres and should be used as a reference. Below is an example of such usage.

To pull a specific postgresql10 image, use this command:

$ docker pull postgres:10-alpine 3.16

To run the Postgresql image in a container using docker, use the following command:

$ docker run -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=<password> \
  -e PGDATA=/var/lib/postgresql/data/pgdata \
  -v <existing path for storing data>:/var/lib/postgresql/data \
  --name my_postgres \
  --network="host" postgres:10-alpine3.16

Note the following portion of the docker run command:

    -e PGDATA=/var/lib/postgresql/data \
    -v <existing path for storing data>:/var/lib/postgresql/data \

The PGDATA environment variable defines the Postgres data storage location within the container. The volume mount command specified with the -v option is required to save the database data on a persistent volume outside of the docker container context.

Once the Postgres database is installed, there is a setup that needs to be done using psql commands.

To perform the setup with Postgres running in a container called, for example, “my_postgres”:

1. Use docker to launch a shell terminal you can use the following docker command:

   Shell command: launch shell terminal (example)

   $  docker exec -it my_postgres bash
   

2. In that shell, invoke the psql utility.

   Shell command: launch psql

   $  psql
   

3. Enter the following psql commands:

   psql commands: setup

   create user fmanager;
   alter user fmanager with encrypted password 'fmanager';
   create database fmanager;
   grant all privileges on database fmanager to fmanager;
   SHOW hba_file;
   \q
   

   The SHOW step should print something like:

                 hba_file              
    /etc/postgresql/10/main/pg_hba.conf
                 (1 row)
   

517.3.2 Run the Elasticsearch docker image

Docker Hub provides a full catalog of Elasticsearch database images (https://hub.docker.com/_/elasticsearch). You can pull an image and run it locally in a docker container.

Instructions for configuring Elasticsearch images and running them in docker containers can be found in https://hub.docker.com/_/elasticsearch and should be used as a reference. Below is an example of such a usage.

To pull a specific Elasticsearch 8.6.0 image, use this command:

$ docker pull elasticsearch:8.6.0

To run the image in a container, use the following docker run command:

$ docker run --name my_elasticsearch \
  -e discovery.type=single-node \
  -v <path to existing directory for data storage>:/usr/share/elasticsearch/data \
  --network="host" elasticsearch:8.6.0

517.4 Run Foretify Manager, Elasticsearch and PostgreSQL images using Kubernetes (k8s) orchestration

The following is an example of a Kubernetes YAML configuration file for defining the following:

• Volumes that are needed for saving configuration files, logs, database persistent data (see above)

• Various container definitions for running Foretify Manager (named fmanager), Postgres (named postgresql), and Elasticsearch (named elasticsearch)

• Defining Foretify Manager environment variables for pointing to Elasticsearch/Postgres instances (FTX_ELASTICSEARCH_HOST, FTX_POSTGRESQL_HOST)

In the example below, localhost:32000 is the URL for the k8s docker registry. The example defines a job that runs all containers on a single pod. This is only one simple use case; a more scalable solution involves running various containers on multiple pods.

This configuration can be applied by using the following command:

$ kubectl apply -f <configuration file>

apiVersion: batch/v1
kind: Job
metadata:
  name: fmanager
spec:
  template:
    metadata:
      labels:
        app.kubernetes.io/name: fmanager
    spec:
      restartPolicy: Never
      volumes:
        - name: fmanager-config
          hostPath:
              path: /path/to/fmanager/config
        - name: dispatcher-config
          hostPath:
              path: /path/to/dispatcher/config
        - name: dispatcher-jobs
          hostPath:
              path: /path/to/dispatcher/jobs
        - name: dispatcher-shared
          hostPath:
              path: /path/to/dispatcher/shared
        - name: logs-dir
          hostPath:
              path: /path/to/log/foretellix
        - name: elastic-data
          hostPath:
              path: /path/to/elasticsearch
        - name: pg-data
          hostPath:
              path: /path/to/postgres

      containers:
      - name: fmanager
        image: localhost:32000/fmanager:test
        imagePullPolicy: Always
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000
          runAsGroup: 1000
        env:
          - name: FTX_LIC_FILE
            value: 30001@172.31.49.28
          - name: dispatcher.ftxLicense
            value: 30001@172.31.49.28
          - name: FTX_ELASTICSEARCH_HOST
            value: 127.0.0.1:9200
          - name: FTX_POSGRESS_SERVER
            value: 127.0.0.1:5432
            ports:
          - containerPort: 8080
            name: fmanager-api
        volumeMounts:
          - mountPath: /opt/foretellix/fmanager/config
            name: fmanager-config
          - mountPath: /opt/foretellix/dispatcher/config
            name: dispatcher-config
          - mountPath: /opt/foretellix/dispatcher/jobs
            name: dispatcher-jobs
          - mountPath: /opt/foretellix/dispatcher/shared
            name: dispatcher-shared
          - mountPath: /var/log/foretellix
            name: logs-dir

      - name: postgresql
        image: localhost:32000/postgres:10-alpine3.16
        env:
          - name: PGDATA
            value: /var/lib/postgresql/data/pgdata
          - name: POSTGRES_USER
            value: postgres
          - name: POSTGRES_PASSWORD
            value: <password>
        volumeMounts:
          - mountPath: /var/lib/postgresql/data
            name: pg-data

      - name: elasticsearch
        image: localhost:32000/elasticsearch:8.6.0
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000
          runAsGroup: 1000
        env:
          - name: discovery.type
            value: single-node
        volumeMounts:
          - mountPath: /usr/share/elasticsearch/data
            name: elastic-data

The following is an example of configuring a service that provides access to Foretify Manager, running with the above pod configuration and using port 8080.

This configuration can be applied by using this command:

$ kubectl apply -f <configuration file>

apiVersion: v1
kind: Service
metadata:
  name: fmanager
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: fmanager
  ports:
  - name: fmanager-api-service-port
    protocol: TCP
    port: 8080
    targetPort: fmanager-api