Our DR plan was a 40-page PDF. The outage never opened it.

Description and use

A small team replaced a 40‑page PDF disaster‑recovery plan with a one‑page, phone‑accessible plan containing clear triggers, RTO/RPO, failover commands, and degraded mode. The new plan passed a drill, uncovered a critical flaw, and is now used in quarterly tests.

Problem

A 40‑page PDF disaster‑recovery plan was unusable during an outage because it was too long, not stored in an accessible place, and lacked actionable steps.

Outcome

The new one‑page plan was successfully used in a drill that completed in 3 h 41 min, revealing a critical flaw (DR bucket in same region). The plan’s brevity and accessibility eliminated the 2‑hour delay caused by the old PDF.

Workflow

• Define a clear trigger (e.g., primary region unreachable for 15+ min or DB restore ETA >2h).
• Specify RTO and RPO with realistic numbers and list call order with phone numbers.
• Document exact failover commands (e.g., aws s3 cp … && psql … && ./smoke-tests.sh).
• Predefine a degraded mode (e.g., pause orders, customer templates).
• Conduct quarterly drills: restore to a scratch instance, run smoke tests, time the process, log results, and create a ticket if the drill fails.

Evidence

During the actual outage, nobody opened it. We opened a chat window and started improvising.

Source

The drill found what the document never would

Source