Deploy, backup and monitor a Django app on two servers

Description and use

The author explains how an OS upgrade caused a five‑hour outage due to process failures. By adopting a per‑release folder layout, atomic symlink switching, and pulling backups from a separate monitoring server with a read‑only key, the deployment became more reliable and observable.

Problem

The author upgraded the Ubuntu OS on the app server from 24.04 to 26.04, causing the site to be unavailable for five hours due to process failures and memory exhaustion.

Outcome

The new deployment and backup strategy prevented a five‑hour outage, reduced memory pressure, and provided automated health reporting that correlates metrics with code changes.

Workflow

• Dump the database
• Create a release worktree and link shared files
• Build a virtualenv if requirements changed
• Run migrations and collectstatic from the new release while the old one serves
• Atomically switch current to the new release
• Restart gunicorns and Daphnes one at a time
• Run a smoke test and revert if any request returns 500

Evidence

The upgrade took 20 minutes. The site came back five hours later.

Source

Every deploy is its own folder: /var/www/html/releases/<sha>/

Source

Backups are pulled nightly at 02:00 UTC and before every deploy. At 03:00 the monitoring box pulls the dumps.

Source

The monitoring box has two SSH keys on the app box, one per directory, and each is locked down in authorized_keys like this: command="rrsync -ro /var/backups/extratime/daily",restrict,from="<monitoring box's WireGuard address>" ssh-ed25519 AAAA...

Source

Exit code 126 means the shell found the program but couldn't execute it.

Source