Skip to content

Deploying

One VM, 138.199.217.95, /srv/evocases-iframe. Images are pulled from ghcr.io/evo-case by a self-hosted runner on the VM, so no SSH key lives in GitHub and no registry credential lives on the box.

Changing the stack definition

The deploy does not put the compose files and scripts on the VM. They are shipped deliberately, from a clean checkout:

./scripts/ship.sh          # or: make ship

It rsyncs the definition and records the shipped commit in .stack-revision. The deploy compares that against the commit it is running from and refuses if they differ, so "merged but never shipped" fails immediately, with that command in the error, rather than three steps later as a confusing compose error.

Why not have the deploy rsync its own checkout? Because then a revert, a stray branch or a bad merge reaches the host the moment anyone presses deploy. Changing what runs should be an act, not a side effect.

Deploy

Normally nothing to do: a push to an application repo builds its images and dispatches the deploy. By hand:

gh workflow run deploy-stage.yml --repo evo-case/infra-evocases-iframe \
  -f be_tag=<sha> -f fe_tag=<sha>

or on the host itself, during an incident:

ssh deploy@<vm> 'cd /srv/evocases-iframe && ./scripts/deploy.sh'

deploy.sh pulls, swaps with --wait, and on failure dumps ps --all plus the last 50 log lines and prints the previous tag to roll back to.

Its file set is not optional:

Overlay Leave it out and…
docker-compose.doppler.yml containers keep the image entrypoint, never run doppler run, and start with no configuration at all
docker-compose.dbaccess.yml the next deploy silently removes developer database access
docker-compose.registry.yml it would build from sources that are not on the VM

docker-compose.override.yml is never loaded here — that file publishes ports and is for laptops.

Tags

Two repositories, two histories, two tags:

BE_TAG=<sha of evocases-iframe-be>
FE_TAG=<sha of evocases-iframe-fe>

A single tag could only ever be right for one of them. Roll back by setting the previous SHA in .env and re-running deploy.sh.

Migrations and seed

docker compose  run --rm migrate                  # prisma migrate deploy
docker compose  --profile seed run --rm be-seed   # bootstrap only

migrate is a separate service rather than a prefix on games-api's command: three apps share this database and all three would race to migrate it on a cold start.

Order matters on a fresh environment

bootstrap.ts creates the first super admin with a password that is a constant in the application repository, and does not print it. Mandatory TOTP is the only thing protecting the account, and it belongs to whoever enrols first.

So: seed, replace the password, sign in and enrol TOTP — and only then publish bo-api.

Config changes

  • Backend (be, shared) — the containers read Doppler at process start. A restart is enough; no rebuild.
  • Frontend (fe, shared) — VITE_* are compiled into the bundle. It needs a rebuild, and restarting instead changes nothing while reporting success.

Backups

evocases-iframe-db-backup.timer, daily, pg_dump to Cloudflare R2 (evocases-iframe-backups). R2_REQUIRED=1 is set, so a silent upload failure is a unit failure.

Restore:

./scripts/db-restore.sh <file>

Check the service, never the timer — a timer is active whenever it is scheduled, which is always:

systemctl show -p Result --value evocases-iframe-db-backup.service          # success
systemctl show -p ExecMainExitTimestampMonotonic --value …-db-backup.service # 0 = never ran

A backup you have not restored is not a backup. Fetch it back out of R2 and load it into a throwaway database.