Health checks

A deploy moves traffic to new containers only once they are healthy. It keeps the old containers until the new release has passed every check.

Three kinds of checks gate a rollout:

• the container's own healthcheck;
• an optional internal HTTP check, which the agent runs from inside the app's network before traffic moves;
• optional public checks of your URLs, after traffic moves.

services:
  web:
    build: .
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:3000/healthz"]
      interval: 5s
      timeout: 3s
      retries: 3
      start_period: 20s
    x-dockflow:
      routes:
        - domains: [shop.example.com]
          port: 3000
      health:
        http:
          path: /healthz
          status: 200-299
          retries: 5
          interval: 2s

version: 1
app: shop
health:
  endpoints:
    - name: home
      url: https://shop.example.com/
environments:
  production: {}

The order of a rolloutCopy Anchor Link

For each service the deploy changes:

1. The new containers start, next to the old ones for the switch and start-first strategies.
2. Dockflow waits until each new container is healthy: its healthcheck passes, or, without one, it has been running for the grace period.
3. The internal HTTP check, if the service declares one, runs against each new container.
4. Traffic moves to the new containers (switch), or the old ones are stopped (start-first).

Once every service is through, and after the after jobs:

5. After health.startup_delay, the public checks of health.endpoints run against your URLs.
6. The old containers are drained and stopped, and the release is committed.

A failure at any step before the commit puts the previous release back. For a routed service, the switch points at the old containers again, which takes about a second. You can change what a failed HTTP check does with on_failure. See Rollouts for the strategies themselves.

The container healthcheckCopy Anchor Link

The Docker healthcheck is the primary signal: a healthcheck: in the Compose file, or a HEALTHCHECK in the image. Dockflow waits until Docker reports the container healthy. It fails the rollout as soon as Docker reports unhealthy.

The rollout of a service also fails when a new container:

• exits;
• restarts three times while Dockflow waits (a crash loop);
• is not healthy within the timeout.

Dockflow computes the timeout from the Compose healthcheck:

start_period + (interval + timeout) × (retries + 1) + 30s

The result is at least 60 seconds. Values you leave out take Docker's defaults (30 s, 30 s, 3 retries). To replace the timeout for every service, set deploy.timeout on the environment:

version: 1
app: shop
environments:
  production:
    deploy:
      timeout: 10m

Services without a healthcheck: graceCopy Anchor Link

A container without a healthcheck counts as healthy once it has been running for x-dockflow.health.grace (default 10s). If it restarts meanwhile, the count starts again.

services:
  worker:
    build: .
    command: ["node", "worker.js"]
    x-dockflow:
      health:
        grace: 20s

For a routed service, running is a weak signal: the process may not answer yet. The plan warns about a routed service with neither a healthcheck nor an internal HTTP check.

Internal HTTP check: x-dockflow.health.httpCopy Anchor Link

Use an internal check for what a container healthcheck cannot test: an image without curl or wget, or a readiness endpoint you don't want Docker to call every few seconds.

The agent joins the app's network for the run and sends GET requests to each new container, by its name. The check runs once the containers are healthy, before traffic moves to them.

services:
  api:
    image: ghcr.io/acme/api:1.4.2
    x-dockflow:
      health:
        http:
          path: /ready
          port: 8080
          status: "204"
          timeout: 2s

Redirects are not followed: a 301 is a status like any other, and the default range accepts it.

The check reaches a service only on one of the app's networks. A service with network_mode: host skips the check, with a warning.

Public checks: health.endpointsCopy Anchor Link

Public checks request full URLs once traffic has moved, through DNS, your edge and TLS, the way your users reach the app. The agent runs them, on the server.

version: 1
app: shop
health:
  startup_delay: 10s
  endpoints:
    - name: home
      url: https://shop.example.com/
    - name: api
      url: https://shop.example.com/api/status
      method: HEAD
      expected_status: 204
      timeout: 10s
      retries: 5
      retry_delay: 3s
environments:
  production: {}
  staging:
    health:
      endpoints:
        - url: https://staging.shop.example.com/
          validate_certs: false

The checks go through public DNS, so a domain behind a CDN such as Cloudflare is checked through it. Certificates are verified against the Mozilla root certificates built into Dockflow.

The checks run in order, and the first that fails stops the run. The old containers are still running during these checks, so a failure switches back to them at once.

Redirects are not followed. An http:// URL on a domain that redirects to HTTPS answers 301 or 308, not 200.

health can be set per environment. There, it replaces the top-level block as a whole, as staging does above.

on_failureCopy Anchor Link

on_failure decides what a failed HTTP check, internal or public, does. Set it in dockflow.yml, top-level or per environment:

dockflow deploy --no-rollback does what fail does, for one run. Use it to look at a failing release before it goes away.

A failure of the rollout itself always rolls back, whatever on_failure says: a container that exits, crash-loops, turns unhealthy or times out. A half-switched service is never a useful state.

After the deploy: the watchdogCopy Anchor Link

Docker restarts a container that exits, but not one that keeps running while its healthcheck fails. An app stuck on a deadlock or a lost database connection stays unhealthy until someone restarts it.

The watchdog, a small container that server init installs, does that restart:

# compose.yml (fragment)
services:
  api:
    image: ghcr.io/acme/api:2.1
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:3000/healthz"]
      interval: 10s
    x-dockflow:
      health:
        restart_after: 1m           # default 2m; 0 never restarts

The watchdog never acts while a run holds the environment's lock, and never on the containers of a rollout in progress: a deploy decides about its own containers.

It needs a healthcheck, in Compose or in the image. Without one, a container is never unhealthy.

The watchdog also:

• brings apps back after a reboot; see secrets and configs;
• finishes or rolls back a run within seconds when the run's agent dies (out of memory, a Docker restart, a reboot), as the next run would; see Interrupted runs.

To go without it, run server init --no-watchdog.

Seeing the checksCopy Anchor Link

Each passing check emits a health.check event with the URL and the status. Failed attempts are debug events: run with --verbose to see every attempt as it happens.

When a run fails, the error names the check, the URL, the last answer and the number of attempts.

dockflow diagnose explains why containers don't stay up: exit codes, crash loops, failed healthchecks with their last output, out-of-memory kills. See Events and Exit codes.