The Cleanup Job Did Exactly What We Told It To

Description and use

A nightly cleanup job deleted 400,000 non‑temporary objects after a prefix rename because the filter failed. Safeguards such as deletion bounds, mandatory dry‑runs, and a delay were added.

Problem

A nightly cleanup job deleted 400,000 non-temporary objects after a storage prefix rename, because the job’s filter no longer matched temporary files and there were no safeguards against excessive deletions.

Outcome

The incident highlighted the need for quantitative limits, mandatory dry‑runs, and a recovery window in destructive automation.

Workflow

• 1. Rename storage prefix during tidy‑up. 2. Job’s filter no longer matches temporary files. 3. Job deletes all matched objects (400k non‑temporary). 4. No failure or warning; monitoring only checks job execution. 5. Post‑incident: add deletion bound, enforce dry‑run, introduce delay between marking and deleting.

Evidence

The job had been deleting temporary artefacts every night for about two years without anyone noticing it existed.

Source

The script did not fail. It did not warn.

Source

Every destructive automation we run now carries three things it did not carry before.

Source