The Cleanup Job Did Exactly What We Told It To

Описание и применение

Автоматическая очистка удалила 400 000 неверных объектов после переименования префикса, так как фильтр перестал работать. Добавлены ограничения, обязательный dry‑run и задержка для восстановления.

Задача

A nightly cleanup job deleted 400,000 non-temporary objects after a storage prefix rename, because the job’s filter no longer matched temporary files and there were no safeguards against excessive deletions.

Результат

The incident highlighted the need for quantitative limits, mandatory dry‑runs, and a recovery window in destructive automation.

Рабочий процесс

• 1. Rename storage prefix during tidy‑up. 2. Job’s filter no longer matches temporary files. 3. Job deletes all matched objects (400k non‑temporary). 4. No failure or warning; monitoring only checks job execution. 5. Post‑incident: add deletion bound, enforce dry‑run, introduce delay between marking and deleting.

Доказательства

The job had been deleting temporary artefacts every night for about two years without anyone noticing it existed.

Источник

The script did not fail. It did not warn.

Источник

Every destructive automation we run now carries three things it did not carry before.

Источник