A temporary job hit its error path and ran 14,000 times in an hour. Nobody had built a ceiling above it.

Описание и применение

Автоматический скрипт без ограничений выполнил 14 000 раз за час, отправив массовые письма и сделав 200 000 API‑запросов. Внедрение единой команды‑выключателя, внешних потолков и идемпотентности остановило проблему за 40 секунд и создало повторяемый шаблон защиты.

Задача

A scheduled automation retried endlessly, causing 14,000 executions in an hour, sending mass emails, making 200,000 API calls, and double‑charging customers.

Результат

The runbook reduced manual intervention time from hours to minutes, prevented mass email and API abuse, and established a repeatable guardrail framework for all automations.

Рабочий процесс

• 1. Add a flag file wrapper that exits if /etc/job.pause exists.
• 2. Enforce external ceilings: max runs per window, max spend per day, dead‑letter after N failures.
• 3. Add idempotency keys to external side effects and guard checks before state changes.
• 4. Deploy a kill switch that can be triggered with a single command.
• 5. After an incident, run a five‑line checklist to identify missing guardrails, assess impact, verify kill switch, and schedule quarterly re‑test.

Доказательства

We stopped it in 40 seconds — because two weeks earlier we'd installed a kill switch.

Источник

A job ID plus an idempotency key on every external side effect, and a check-before-act guard on stateful steps, turn re-runs into log lines instead of customer calls.

Источник