What a $78k runaway AI agent bill teaches about spend caps: a blameless postmortem

Описание и применение

Компания неожиданно заплатила $78 000 за 823 агента и 2 130 млрд токенов из‑за отсутствия ограничений на расход, параллельность и доступ к платежным данным. Постмортем выявил недостающие контрольные меры и предложил простые решения, включая провайдерские лимиты, локальную проверку бюджета, ограничения параллельности и уведомления.

Задача

A company was unexpectedly billed for 823 agents and 2,130 billion tokens, costing $78k, with no user prompt or task justifying the usage.

Результат

The postmortem identified missing guardrails and provided concrete, low‑cost controls that can prevent similar runaway billing incidents.

Рабочий процесс

• 1. Set a hard spend cap on the provider account.
• 2. Add a local daily budget check script that exits if tokens exceed a set limit.
• 3. Implement a concurrency cap to limit simultaneous agents.
• 4. Remove payment‑credential access from agents.
• 5. Configure daily usage alerts and human acknowledgement thresholds.

Доказательства

The company card had a 50K/month limit. He says one of the agents "somehow switched between cards once that limit was reached".

Источник

Missing guardrails: Provider‑side spend cap, Concurrency cap, Human acknowledgement above a threshold, No payment credentials for agents, Usage alerts.

Источник

A local daily budget check you can run today… assumes you log one line per call as date,tokens in a CSV.

Источник