I Built a Disaster Recovery Tool Because Row Counts Lied to Me

Description and use

Ark was built to verify real recoverability from backups using AWS as a pilot‑light recovery environment. The first drill finished in 1 min 58 s, meeting a 15‑min RTO target.

Problem

Backups may appear successful (e.g., matching row counts) but may not actually recover the full state, especially for data with non-deterministic metadata like Git objects.

Outcome

Ark enabled a fully automated recovery drill that completed in under 2 minutes, meeting a 15‑minute RTO target and demonstrating that backups were truly restorable.

Workflow

• Collect database dumps, Docker volumes, config files
• Upload backup files to S3
• Upload SHA‑256 manifest last to mark completion
• Terraform provisions VPC, security groups, launch template
• On drill, launch fresh EC2 instance
• Download latest completed backup
• Verify checksums
• Restore database and volumes
• Start application
• Run readiness check to confirm application is functional

Evidence

The first recovery drill achieved a 1m 58s RTO using a real backup.

Source

Backup files → upload ↓ Verification ↓ Manifest → upload last

Source