The Cleanup Job Did Exactly What We Told It To

说明与用途

一次夜间清理作业因前缀重命名导致过滤器失效，错误删除了40万非临时对象。随后加入了删除上限、强制预演和延迟恢复等安全措施。

问题

A nightly cleanup job deleted 400,000 non-temporary objects after a storage prefix rename, because the job’s filter no longer matched temporary files and there were no safeguards against excessive deletions.

结果

The incident highlighted the need for quantitative limits, mandatory dry‑runs, and a recovery window in destructive automation.

工作流

• 1. Rename storage prefix during tidy‑up. 2. Job’s filter no longer matches temporary files. 3. Job deletes all matched objects (400k non‑temporary). 4. No failure or warning; monitoring only checks job execution. 5. Post‑incident: add deletion bound, enforce dry‑run, introduce delay between marking and deleting.

证据

The job had been deleting temporary artefacts every night for about two years without anyone noticing it existed.

来源

The script did not fail. It did not warn.

来源

Every destructive automation we run now carries three things it did not carry before.

来源