The apply we interrupted and the resources state never heard about

Description and use

An interrupted Terraform apply left resources in the cloud but not in state, causing duplicates and unexpected costs. Implementing dynamic timeouts, safe cancellation handling, and a post‑interruption refresh/diff/import workflow resolves the issue.

Problem

An interrupted Terraform apply left resources partially created, causing orphan resources and state inconsistencies, leading to unexpected costs and failed subsequent runs.

Outcome

The new process prevents orphan resources, eliminates duplicate NAT gateways, and keeps costs predictable by ensuring state reflects reality after interruptions.

Workflow

• 1. Run Terraform apply with dynamic timeout and cancellation‑safe step marking.
• 2. If apply is interrupted, run terraform refresh to sync state with actual resources.
• 3. Execute terraform plan to identify differences between state and real resources.
• 4. Import any existing resources missing from state using terraform import.
• 5. Re‑run terraform apply to apply remaining changes.
• 6. Weekly: run a sweep script that lists resources with the workspace tag and compares them against state to detect forgotten resources.

Evidence

Terraform does not roll anything back when you kill it. Resources that were created before the process died exist in the cloud; whether they exist in state depends on whether the state write completed.

Source

We also run a weekly sweep that lists resources carrying our workspace tag and compares them against state, which is how you find things Terraform has forgotten it owns.

Source