I Built a Disaster Recovery Tool Because Row Counts Lied to Me

Описание и применение

Создан инструмент Ark для проверки реальной возможности восстановления из резервных копий, использующих AWS как среду восстановления. Первый тестовый запуск завершился за 1 мин 58 сек, удовлетворив целевой RTO 15 мин.

Задача

Backups may appear successful (e.g., matching row counts) but may not actually recover the full state, especially for data with non-deterministic metadata like Git objects.

Результат

Ark enabled a fully automated recovery drill that completed in under 2 minutes, meeting a 15‑minute RTO target and demonstrating that backups were truly restorable.

Рабочий процесс

• Collect database dumps, Docker volumes, config files
• Upload backup files to S3
• Upload SHA‑256 manifest last to mark completion
• Terraform provisions VPC, security groups, launch template
• On drill, launch fresh EC2 instance
• Download latest completed backup
• Verify checksums
• Restore database and volumes
• Start application
• Run readiness check to confirm application is functional

Доказательства

The first recovery drill achieved a 1m 58s RTO using a real backup.

Источник

Backup files → upload ↓ Verification ↓ Manifest → upload last

Источник