A temporary job hit its error path and ran 14,000 times in an hour. Nobody had built a ceiling above it.

说明与用途

一个没有限制的自动化脚本在一小时内执行了14,000次，发送了大量邮件并进行了200,000次API调用。通过实施单命令停止开关、外部上限和幂等性，在40秒内停止了它，并建立了可重复的防护框架。

问题

A scheduled automation retried endlessly, causing 14,000 executions in an hour, sending mass emails, making 200,000 API calls, and double‑charging customers.

结果

The runbook reduced manual intervention time from hours to minutes, prevented mass email and API abuse, and established a repeatable guardrail framework for all automations.

工作流

• 1. Add a flag file wrapper that exits if /etc/job.pause exists.
• 2. Enforce external ceilings: max runs per window, max spend per day, dead‑letter after N failures.
• 3. Add idempotency keys to external side effects and guard checks before state changes.
• 4. Deploy a kill switch that can be triggered with a single command.
• 5. After an incident, run a five‑line checklist to identify missing guardrails, assess impact, verify kill switch, and schedule quarterly re‑test.

证据

We stopped it in 40 seconds — because two weeks earlier we'd installed a kill switch.

来源

A job ID plus an idempotency key on every external side effect, and a check-before-act guard on stateful steps, turn re-runs into log lines instead of customer calls.

来源