Dockflow

Uploads

Send files and directories to an app volume or a path on the server, incrementally and with exact rollbacks.

uploads: sends local files or directories to the server on every deploy. For example: a static site into a volume that a web server container serves, or a systemd unit into /etc/systemd/system.

Only the files that changed travel, and each file is replaced atomically. Every release records what it uploaded, so a rollback puts the previous files back.

A static site served from a volume

compose.yml
services:
  web:
    image: nginx:1.27-alpine
    volumes:
      - site:/usr/share/nginx/html:ro
    healthcheck:
      test: ["CMD", "wget", "-q", "--spider", "http://localhost/"]
    x-dockflow:
      routes:
        - domains: [docs.example.com]
          port: 80
volumes:
  site: {}
dockflow.yml
version: 1
app: docs
uploads:
  - src: public
    dest: volume:site
    mirror: true
    exclude: ["*.map", ".DS_Store"]
environments:
  production: {}

On each deploy, the content of public/ becomes the content of the site volume. With mirror: true, files you deleted locally are deleted from the volume too.

The container's image never changes. A deploy where only the site changed uploads the changed files and leaves the web container running.

A file on the host

dockflow.yml
version: 1
app: docs
uploads:
  - src: deploy/backup.timer
    dest: /etc/systemd/system/docs-backup.timer
    mode: "644"
    owner: "0:0"
hooks:
  after_deploy:
    - host: sudo systemctl daemon-reload
environments:
  production: {}

A host hook can act on the uploaded file once the deploy is done.

Options

KeyTypeDefaultDescription
srcpathrequiredA file or a directory of the project, relative to the project directory
deststringrequiredvolume:<name>, a volume of the Compose file, or an absolute path on the server
mirrorbooleanfalseDelete, at the destination, the files that src no longer has. For a directory source only
excludelist of patterns[]Paths of src to leave out, in .dockerignore syntax, with ! to re-include
modeoctal stringeach file's local modeThe mode of every file written, such as "644". Quote it
owner"uid" or "uid:gid"unchangedThe numeric owner of every file written. Names such as www-data are refused by dockflow validate

Two entries cannot share a dest.

uploads: can also be set inside an environment. There, it replaces the top-level list for that environment.

Destinations

destWhat it is
volume:<name>A volume declared under the top-level volumes: of the Compose file, by its key (site, not docs-production_site). The source should be a directory: its content becomes the volume's content. Dockflow creates the volume if it does not exist yet
/absolute/pathA file or a directory on the server. Missing parent directories are created

A volume: name that the Compose file does not declare stops dockflow plan and dockflow deploy with exit code 3. dockflow validate checks only the form of dest.

exclude is the only filter: .gitignore and .dockerignore do not apply to uploads. An upload of . sends .git unless you exclude it.

How an upload works

  1. Templates are rendered. A file inside src that is listed under templates: is uploaded rendered.
  2. The CLI describes the source. It hashes every file (after exclude) into a manifest: each path with its SHA-256 and mode. The release records the manifest's digest, so dockflow plan shows an upload for a destination exactly when its files changed.
  3. Only missing files travel. The server keeps a content-addressed store of the app's files. The CLI asks which contents the server lacks and sends only those. Each file is checked against its hash on arrival. A 2 GB directory where one file changed sends one file.
  4. A helper container applies the manifest. The agent starts a short-lived container that mounts the store and the destination (the volume, or the parent directory of a host path). Each changed file is written to a temporary file next to its target, then renamed over it. A reader sees either the old file or the new one, never a partial one. With mirror, files the manifest does not list are deleted, then empty directories.

The helper never follows a symbolic link out of the destination. A container that can write to the destination could replace a directory with a link to elsewhere on the server: the upload then fails instead of writing there. Of /var/lib/dockflow, the helper mounts only the store, read-only, the upload manifests, and an empty directory for the files it replaces.

The swap is atomic per file, not for the whole directory. While the upload runs, a container serving the volume can see some new files next to some old ones.

In the plan, uploads run after networks and volumes are created and before the jobs and services. A before job can therefore read what was uploaded.

Rollbacks and failed deploys

Before the helper overwrites or deletes a file, it saves that file into the store. Every upload can therefore put its destination back exactly as it was, including removing files that did not exist before.

  • A failed deploy that rolls back also restores its uploads, newest first.
  • dockflow rollback applies the manifests recorded in the release it restores, so the files come back with the containers. Their contents stay in the store as long as that release is in retention.

Removing an entry from uploads: does not delete what it uploaded: the files stay at the destination.

An app with only uploads

An app does not need any service. Without a Compose file, or with an empty one, a dockflow.yml with uploads: is a complete app. Each deploy sends the files and records a release, which can be rolled back.

dockflow.yml
version: 1
app: maven
uploads:
  - src: repository
    dest: /srv/maven
    mirror: true
    exclude: [".git", "*.tmp"]
environments:
  production: {}

Here a web server already on the host serves /srv/maven. Later deploys send only the artifacts that changed, and mirror: true removes deleted ones.

Disk space

The store keeps the contents of every release in retention, and the files that uploads replaced. Large trees that change often take space on the server until their releases leave retention. See Cleanup.

Edit on GitHub

On this page