Our DR plan was a 40-page PDF. The outage never opened it.

Описание и применение

Небольшая команда заменила 40‑страничный PDF‑план DR на одностраничный, доступный на телефоне, с четкими шагами и тестами. План прошёл проверку в реальном сценарии, выявив критическую ошибку, и теперь используется в ежеквартальных учениях.

Задача

A 40‑page PDF disaster‑recovery plan was unusable during an outage because it was too long, not stored in an accessible place, and lacked actionable steps.

Результат

The new one‑page plan was successfully used in a drill that completed in 3 h 41 min, revealing a critical flaw (DR bucket in same region). The plan’s brevity and accessibility eliminated the 2‑hour delay caused by the old PDF.

Рабочий процесс

• Define a clear trigger (e.g., primary region unreachable for 15+ min or DB restore ETA >2h).
• Specify RTO and RPO with realistic numbers and list call order with phone numbers.
• Document exact failover commands (e.g., aws s3 cp … && psql … && ./smoke-tests.sh).
• Predefine a degraded mode (e.g., pause orders, customer templates).
• Conduct quarterly drills: restore to a scratch instance, run smoke tests, time the process, log results, and create a ticket if the drill fails.

Доказательства

During the actual outage, nobody opened it. We opened a chat window and started improvising.

Источник

The drill found what the document never would

Источник