Infrastructure Maintenance

This guide is designed to help you upgrade your self-hosted Tines infrastructure and its external components. Before attempting any operations in this guide, we strongly urge you to take backups of your database. We also suggest that you do not upgrade Tines versions while making core infrastructure changes.

Shutdown and Startup of Tines Services 

If you are performing maintenance or migrations on the underlying infrastructure hosting Tines, follow the preferred order below to gracefully shut down. Following these steps will cause downtime for your Tines tenant. This order is applicable to Docker Compose, ECS, and Kubernetes. For each step, reduce the number of running instances to zero.

Before starting the shutdown process, review the section "Checking tines-sidekiq queue size." If samples show non-zero values that are not reducing, you should scale the tines-sidekiq service before starting the shutdown.

Scale-down order:

  1. nginx or other proxy service (if applicable)

  2. tines-app

  3. tines-sidekiq (Ensure the queue size reaches near 0 and is not increasing during sampling before proceeding).

  4. tines-command-runner

  5. Redis/Valkey

  6. Postgres

  7. pypi-server

Examples of scaling down by platform: 

Docker Compose:

docker compose stop tines-app

ECS:

aws ecs update-service --cluster tines --service tines-app --desired-count 0

Kubernetes:

kubectl scale deployment tines-app --replicas=0

Checking tines-sidekiq Queue Size 

Run the following Rails command from within a tines-sidekiq container to check the queue size. Run this command multiple times over a 2–5 minute window to ensure the volume is decreasing.

bundle exec rails runner "puts JSON.pretty_generate(Sidekiq::Queue.all.map { |q| { name: q.name, size: q.size, latency: q.latency.round(2), paused: q.paused? } })"

Accessing the container:

aws ecs execute-command --cluster \
--task \
--container tines-sidekiq \
--interactive \
--command "/bin/bash

Migrating to a New Database Instance 

Before performing operations on your PostgreSQL instance, ensure you have recent backups and snapshots in place. Migration time varies based on data volume and network speed. Do not run this process from a laptop or any machine that may enter sleep mode.

Prerequisites:

  1. All Tines services shut down except Postgres.

  2. A host with pg_dump and pg_restore installed.

  3. Network connectivity between source and destination databases.

  4. Adequate local storage for the dump file.

  5. Existing DATABASE_NAME and DATABASE_HOST configuration details.

Export 

pg_dump -h <existing_db_host> \
        -U <username> \
        -d <database_name> \
        -Fc \
        -v \
        -f tines_migration_snapshot.dump

Import 

pg_restore -h <new_db_host> \
           -U <username> \
           -d <database_name> \
           -v \
           tines_migration_snapshot.dump

Migrating to a New Redis/Valkey Instance 

Redis/Valkey is an in-memory, transient datastore that does not require manual migrations or backups. To make cluster changes (such as adding nodes), follow the standard shutdown and startup procedure listed above.

Changing DNS for External Services 

Changing the DNS for external services (e.g., Redis/Valkey or Postgres) should never be performed while Tines is running.

Update the following values in your configuration:

Docker Compose and ECS (.env):

REDIS_URL=redis://<new_dns_host>:6379/1
DATABASE_HOST=<new_dns_host>

Helm (values.yaml):

tinesApp:
  databaseConfiguration:
    DATABASE_HOST: <new_dns_host>
  redisConfiguration:
    REDIS_URL: redis://<new_dns_host>:6379/1

Note: Before restarting Tines, ensure the new DNS entry has propagated and is accessible from your infrastructure. At a minimum, wait for the duration of the DNS TTL and flush any local DNS caches if necessary.

Was this helpful?