I Built a Disaster Recovery Tool Because Row Counts Lied to Me

说明与用途

Ark 工具用于验证从备份中真正恢复的能力，使用 AWS 作为恢复环境的“灯塔”模式。第一次演练在 1 分 58 秒内完成，满足 15 分钟的 RTO 目标。

问题

Backups may appear successful (e.g., matching row counts) but may not actually recover the full state, especially for data with non-deterministic metadata like Git objects.

结果

Ark enabled a fully automated recovery drill that completed in under 2 minutes, meeting a 15‑minute RTO target and demonstrating that backups were truly restorable.

工作流

• Collect database dumps, Docker volumes, config files
• Upload backup files to S3
• Upload SHA‑256 manifest last to mark completion
• Terraform provisions VPC, security groups, launch template
• On drill, launch fresh EC2 instance
• Download latest completed backup
• Verify checksums
• Restore database and volumes
• Start application
• Run readiness check to confirm application is functional

证据

The first recovery drill achieved a 1m 58s RTO using a real backup.

来源

Backup files → upload ↓ Verification ↓ Manifest → upload last

来源