Run Script Setup

We offer the Run Script feature for self-hosted customers and for cloud customers using tunnels. For general details on how to use this, please refer to the documentation here.

Overview 

The self-hosted configuration for this feature requires the configuration of a seperate application called tines-command-runner.

Our Docker based Run Scripts implementation leverages two containers;

  1. tines-command-runner

  2. pypi-server(https://github.com/pypiserver/pypiserver)

The tines-command-runner is where run scripts will execute.

Usage of pypi-server is optional. The pypi-server acts as a local python package index, making the installation of packages efficient and easy to configure as needed. pypi-server can be overridden to use a custom index via environment variables to rely on a different package index. See environment variables section for further details.

Environment variables 

You can override certain settings using environment variables.

The environment variables have defaults if not populated but can be tweaked if you want to run your own Python Package Index and point to it.

  • PIP_INDEX_URL - Specifies the primary, default Python Package index. Default: https://pypi.org/simple. Can be set to a custom index.

  • PIP_EXTRA_INDEX_URL - Allows the customer to specify a fallback Python package index. Default: pypi-server , the provided local pypi server

  • NO_PIP_INDEX - Disables any index, even if they are specified by PIP_INDEX_URL or PIP_EXTRA_INDEX_URL, instead relying entirely on the packages included already in the container.

By default we specify the extra index as the pypi-server that runs alongside the command runner.

  • PIP_EXTRA_INDEX_URL=http://pypi-server:8080/simple/

  • TRUSTED_HOST=pypi-server

  • LOG_LEVEL - Configures the logging level for the Python harness. Set to `DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL` (defaults to `INFO`). Set `LOG_LEVEL=DEBUG` to enable debug logging for troubleshooting. Logs are written to /tmp/tcr-logs/harness-{environment_id}-{stdout,stderr}.log.

For timeouts, you can configure:

  • RUN_SCRIPT_MAX_TIMEOUT - see more information below.

Pre-Downloaded Python Packages 

The below common packages are pre-downloaded in the image for efficient access. These packages are available even if NO_PIP_INDEX=true. More packages may be added to this list in the future, versions may change as well.

annotated-types
anyio
beautifulsoup4
boto
boto3
click
cryptography
django
fastapi
flask
grpcio
grpcio-reflection
grpcio-tools
h11
idna
jupyter
lxml
matplotlib
networkx
nltk
numpy
openpyxl
pandas
paramiko
plotly
protobuf
pyarrow
pyopenssl
pydantic
pydantic_core
pytest
pytz
requests
scikit-learn
scipy
seaborn
setuptools
sniffio
sqlalchemy
starlette
statsmodels
sympy
typing_extensions
uvicorn
xlrd

Note: All transitive dependencies required by the above packages are also automatically downloaded and available for use. This means the actual number of available packages is larger than this explicit list, as each package brings in its own required dependencies during the build process.

Step 1. Prepare the tines-command-runner Docker image​  

To make things a little easier, we mirror the image from Docker Hub into an ECR repository:

aws --profile test ecr create-repository --repository-name tines-command-runner

# Replace this with the address of the registry output in the previous command:
REGISTRY=306378194054.dkr.ecr.eu-west-1.amazonaws.com

aws ecr get-login-password --region eu-west-1 | \
  docker login --username AWS --password-stdin $REGISTRY

TCR_IMAGE=tines-command-runner:latest

docker pull tines/$TCR_IMAGE
docker tag tines/$TCR_IMAGE $REGISTRY/$TCR_IMAGE
docker push $REGISTRY/$TCR_IMAGE

Step 2. Update the ECS task definitions​  

To make it easy to access, we simply deploy the tines-command-runner container alongside the existing containers by adding it to the tines-sidekiq task. You can update the JSON payload in the AWS ECS UI by locating the latest task definition and include the following container definition:

{
  "name": "tines-command-runner",
  "image": "$REGISTRY/tines-command-runner:latest",
  "environment": [
    {
      "name": "PORT",
      "value": "4400"
    }
  ],
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-group": "tines",
      "awslogs-region": "$AWS_REGION",
      "awslogs-stream-prefix": "tines"
    }
  },
  "portMappings": [
    {
      "containerPort": 4400
    }
  ]
}

Step 3. Set a new environment variable 

You also need to set the following environment variable on the Tines containers : TINES_COMMAND_RUNNER_HOST=localhost. This should be done on your .env file or task definition.

Step 4. Restart the services​  

aws ecs update-service --cluster <cluster_name> --service tines-sidekiq --force-new-deployment

Run Script Timeout Configuration 

The default timeout for scripts is 60 seconds. For self-hosted deployments, you can override this by setting the RUN_SCRIPT_MAX_TIMEOUT environment variable on your .env file or on the tines-sidekiq application.

This should be done on your .env file or task definition.

.env file

# Other tines-app env vars
RUN_SCRIPT_MAX_TIMEOUT=90

AWS Fargate (add to tines-sidekiq task definition):

{
  "environment": [
    {
      "name": "RUN_SCRIPT_MAX_TIMEOUT",
      "value": "90"
    }
  ]
}
Was this helpful?