Our DR plan was a 40-page PDF. The outage never opened it.

说明与用途

小团队将40页PDF灾难恢复计划替换为一页手机可访问的计划，包含清晰的触发条件、RTO/RPO、故障转移命令和降级模式。新计划通过演练，发现关键缺陷，并用于季度测试。

问题

A 40‑page PDF disaster‑recovery plan was unusable during an outage because it was too long, not stored in an accessible place, and lacked actionable steps.

结果

The new one‑page plan was successfully used in a drill that completed in 3 h 41 min, revealing a critical flaw (DR bucket in same region). The plan’s brevity and accessibility eliminated the 2‑hour delay caused by the old PDF.

工作流

• Define a clear trigger (e.g., primary region unreachable for 15+ min or DB restore ETA >2h).
• Specify RTO and RPO with realistic numbers and list call order with phone numbers.
• Document exact failover commands (e.g., aws s3 cp … && psql … && ./smoke-tests.sh).
• Predefine a degraded mode (e.g., pause orders, customer templates).
• Conduct quarterly drills: restore to a scratch instance, run smoke tests, time the process, log results, and create a ticket if the drill fails.

证据

During the actual outage, nobody opened it. We opened a chat window and started improvising.

来源

The drill found what the document never would

来源