How I deploy, back up and watch a Django app on two servers

说明与用途

升级操作系统后，Python 虚拟环境路径错误导致所有进程退出，网站宕机五小时。新的部署流程创建独立发布文件夹，原子切换当前符号链接，一次重启一个进程，并执行烟雾测试。备份现在从另一个服务器以只读密钥拉取，并使用 age 加密。

问题

The site went down for five hours after an OS upgrade because the virtualenvs’ Python interpreter path was invalid, causing all processes to exit and restart repeatedly.

结果

Deploys are now atomic and safe, backups are secure and encrypted, and monitoring detects and recovers from process failures quickly, reducing downtime.

工作流

• Dump the database
• Create release worktree and link shared files
• Build virtualenv if requirements changed
• Run migrations and collectstatic from new release
• Atomically switch current symlink
• Restart gunicorns and Daphnes one at a time
• Run smoke test and rollback if needed

证据

Deploys used to be git reset --hard in place, and a running process could lazily import a module from the new code while the rest of it was still the old code.

来源

A deploy: 1) dumps the database 2) creates the release worktree and links the shared files 3) builds a venv only if requirements.txt changed 4) runs migrations and collectstatic from the new release while the old one serves 5) switches current with ln -sfn and mv -T, which is atomic 6) restarts the two gunicorns and the two Daphnes one at a time, so one of each is always up 7) runs a smoke test that makes real requests, and switches current back if one returns a 500

来源

Postgres is dumped nightly at 02:00 UTC and before every deploy. At 03:00 the monitoring box pulls the dumps.

来源

Exit code 126 means the shell found the program but couldn't execute it. My reading is that the virtualenvs' Python was the one 24.04 shipped, and 26.04 replaced it with Python 3.14, so every script's interpreter line pointed at something that was gone.

来源