How I deploy, back up and watch a Django app on two servers

Description and use

The site was down for five hours after an OS upgrade because the virtualenv Python path was invalid, causing all processes to exit. The new deployment workflow creates a separate release folder, atomically switches the current symlink, restarts processes one at a time, and runs a smoke test. Backups are now pulled from a separate server with a read‑only key and encrypted with age.

Problem

The site went down for five hours after an OS upgrade because the virtualenvs’ Python interpreter path was invalid, causing all processes to exit and restart repeatedly.

Outcome

Deploys are now atomic and safe, backups are secure and encrypted, and monitoring detects and recovers from process failures quickly, reducing downtime.

Workflow

• Dump the database
• Create release worktree and link shared files
• Build virtualenv if requirements changed
• Run migrations and collectstatic from new release
• Atomically switch current symlink
• Restart gunicorns and Daphnes one at a time
• Run smoke test and rollback if needed

Evidence

Deploys used to be git reset --hard in place, and a running process could lazily import a module from the new code while the rest of it was still the old code.

Source

A deploy: 1) dumps the database 2) creates the release worktree and links the shared files 3) builds a venv only if requirements.txt changed 4) runs migrations and collectstatic from the new release while the old one serves 5) switches current with ln -sfn and mv -T, which is atomic 6) restarts the two gunicorns and the two Daphnes one at a time, so one of each is always up 7) runs a smoke test that makes real requests, and switches current back if one returns a 500

Source

Postgres is dumped nightly at 02:00 UTC and before every deploy. At 03:00 the monitoring box pulls the dumps.

Source

Exit code 126 means the shell found the program but couldn't execute it. My reading is that the virtualenvs' Python was the one 24.04 shipped, and 26.04 replaced it with Python 3.14, so every script's interpreter line pointed at something that was gone.

Source