Jobs and migrations

A job is a service that runs to completion instead of staying up: a database migration, a cache warm-up, an index rebuild. Dockflow runs it as a one-off container on the new release's image and streams its output. If the job fails, the deploy stops or rolls back.

services:
  migrate:
    build: .
    command: ["npm", "run", "db:migrate"]

  web:
    build: .
    depends_on:
      migrate:
        condition: service_completed_successfully

web waits for migrate to complete successfully. That makes migrate a job that runs before the services, as docker compose up would run it. The same file works locally with Compose.

Declaring a jobCopy Anchor Link

There are two ways, and one file can use both:

services:
  warm-cache:
    build: .
    command: ["npm", "run", "cache:warm"]
    x-dockflow:
      run: after

A job takes the rest of its spec from Compose as written: image or build, command, environment, networks, volumes. It runs once, with restart: no, and is never kept as a running service.

When jobs runCopy Anchor Link

Jobs run in a dockflow deploy that changes services, that is, one that creates, updates or removes a service, or changes routes. They don't run:

• in a deploy whose plan is empty, or that only changes uploaded files;
• in dockflow rollback: a rollback does not replay old migrations;
• in restart, stop, scale or destroy.

A change to the job alone does not make a plan. In practice, a job usually shares the image of a service (build: . on both), so a code change updates that service and the job runs with it.

OrderCopy Anchor Link

Within a deploy:

1. Networks, volumes and uploads are created or updated, so a job can read uploaded files.
2. The services and the before jobs run in depends_on order. A before job runs once the services it depends on are up (a migration after its database), and before the services that depend on it. A before job that depends on nothing runs before every service. before_deploy hooks of the exec kind run right before the first service changes.
3. The after jobs run, once every service has switched.
4. The public checks run, then the release is committed.

Jobs of the same kind run in depends_on order, and alphabetically when nothing orders them.

A service that a before job depends on is created or updated first, and is healthy when the job starts. On the very first deploy of an app, a migration that depends on its database (condition: service_healthy) runs once the database is up. Services that depend on the job start after it.

Output, timeout and failureCopy Anchor Link

Each line the job writes is a job.log event, shown as migrate │ <line>. A job.finished event gives its exit code. The container is removed once it has exited.

The timeout is the environment's deploy.timeout, or 30 minutes when it is not set:

version: 1
app: shop
environments:
  production:
    deploy:
      timeout: 20m

deploy.timeout also replaces the health timeout of every service; see Health checks.

on_failure does not apply to jobs: a failed job always stops or rolls back the deploy.

Refused combinationsCopy Anchor Link

dockflow validate refuses:

Example: database migrationsCopy Anchor Link

A web app with Postgres, migrating the schema before each new version starts:

services:
  migrate:
    build: .
    command: ["npm", "run", "db:migrate"]
    environment:
      DATABASE_URL: postgres://shop:${DB_PASSWORD}@db:5432/shop
    depends_on:
      db:
        condition: service_healthy

  web:
    build: .
    environment:
      DATABASE_URL: postgres://shop:${DB_PASSWORD}@db:5432/shop
    depends_on:
      migrate:
        condition: service_completed_successfully
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:3000/healthz"]
    x-dockflow:
      routes:
        - domains: [shop.example.com]
          port: 3000

  warm-cache:
    build: .
    command: ["npm", "run", "cache:warm"]
    x-dockflow:
      run: after

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: shop
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: shop
    volumes: [pgdata:/var/lib/postgresql/data]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U shop -d shop"]
      interval: 5s
    x-dockflow:
      stateful: true

volumes:
  pgdata: {}

DB_PASSWORD comes from the PRODUCTION_DB_PASSWORD secret. On a deploy of a new version, the plan reads:

1. job.run migrate: the migration runs on the new image, against the running database;
2. service.update web: the new web containers start and become healthy, and traffic switches to them;
3. job.run warm-cache: the cache is warmed with the new code.

depends_on: db on migrate keeps the file working with docker compose up, where Compose starts db first.

Migrations that survive a rollbackCopy Anchor Link

A deploy rolls back containers, never data:

• If migrate fails, nothing has switched, and the schema is whatever the migration left.
• If migrate succeeds and the new web containers then fail their checks, or the agent dies, the deploy goes back to the previous code on the new schema.
• dockflow rollback runs no job, so it also leaves the previous code on the new schema.
• While migrate runs, and for a moment after it, the previous web containers keep serving on the new schema anyway.

So write each migration so that the previous release still works on the schema it leaves. Do it in two steps, expand then contract:

1. Expand, in one release: add what the new code needs (new tables, new columns that are nullable or have a default, new indexes), and keep everything the previous code reads and writes. The new code writes both the old and the new shape when they differ.
2. Contract, in a later release, once nothing runs the old code any more: drop the old columns and tables, and add the constraints the expand step could not.

A rename is an expand (add the new column, copy, write both) followed by a contract (drop the old one). With this rule, every rollback lands on code that works with the schema in place. A failed deploy never needs a database restore.

To run a command by hand with a service's spec, such as a console or a one-time script, use dockflow run; see Day-2 operations.

Workers and recurring tasksCopy Anchor Link

Jobs run during deploys only. Dockflow has no scheduler of its own: nothing runs on a timetable on the server. Background work takes one of two forms.

A worker is a long-running service without routes, such as a queue consumer. It gets the start-first strategy: the new container starts and becomes healthy before the old one stops, so for a moment both run. A worker that must never run twice, such as one that runs its own scheduler (supercronic, node-cron), asks for stop-first:

services:
  worker:
    build: .
    command: ["npm", "run", "worker"]
    deploy:
      replicas: 2

  scheduler:
    build: .
    command: ["supercronic", "/app/crontab"]
    x-dockflow:
      rollout:
        strategy: stop-first
      health:
        grace: 10s

A recurring task can run from your CI on a schedule, with dockflow run. It starts a one-off container with the service's spec and the image the service currently runs, and exits with the command's exit code:

on:
  schedule: [{ cron: "0 4 * * *" }]
jobs:
  report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: curl -fsSL https://raw.githubusercontent.com/Shawiizz/dockflow/main/install.sh | sh
      - run: dockflow run production web -- npm run report:nightly
        env:
          DOCKFLOW_SECRETS: ${{ toJSON(secrets) }}

run reads dockflow.yml and the Compose files. The CI job therefore checks out the repository and gets every secret they use, as a deploy does. See Deploying from CI.