Deploy, backup and monitor a Django app on two servers

说明与用途

作者说明了操作系统升级导致网站停机五小时的原因，并通过采用每个发布单独文件夹、原子 symlink 切换以及从单独监控服务器拉取备份（只读密钥）来提高部署可靠性和可观测性。

问题

The author upgraded the Ubuntu OS on the app server from 24.04 to 26.04, causing the site to be unavailable for five hours due to process failures and memory exhaustion.

结果

The new deployment and backup strategy prevented a five‑hour outage, reduced memory pressure, and provided automated health reporting that correlates metrics with code changes.

工作流

• Dump the database
• Create a release worktree and link shared files
• Build a virtualenv if requirements changed
• Run migrations and collectstatic from the new release while the old one serves
• Atomically switch current to the new release
• Restart gunicorns and Daphnes one at a time
• Run a smoke test and revert if any request returns 500

证据

The upgrade took 20 minutes. The site came back five hours later.

来源

Every deploy is its own folder: /var/www/html/releases/<sha>/

来源

Backups are pulled nightly at 02:00 UTC and before every deploy. At 03:00 the monitoring box pulls the dumps.

来源

The monitoring box has two SSH keys on the app box, one per directory, and each is locked down in authorized_keys like this: command="rrsync -ro /var/backups/extratime/daily",restrict,from="<monitoring box's WireGuard address>" ssh-ed25519 AAAA...

来源

Exit code 126 means the shell found the program but couldn't execute it.

来源