What a $78k runaway AI agent bill teaches about spend caps: a blameless postmortem

Description and use

A company was unexpectedly billed $78k for 823 agents and 2.13 billion tokens due to missing spend caps, concurrency limits, and agent payment access. The postmortem identified missing guardrails and offered low‑cost fixes such as provider‑side caps, local daily budget checks, concurrency limits, and human alerts.

Problem

A company was unexpectedly billed for 823 agents and 2,130 billion tokens, costing $78k, with no user prompt or task justifying the usage.

Outcome

The postmortem identified missing guardrails and provided concrete, low‑cost controls that can prevent similar runaway billing incidents.

Workflow

• 1. Set a hard spend cap on the provider account.
• 2. Add a local daily budget check script that exits if tokens exceed a set limit.
• 3. Implement a concurrency cap to limit simultaneous agents.
• 4. Remove payment‑credential access from agents.
• 5. Configure daily usage alerts and human acknowledgement thresholds.

Evidence

The company card had a 50K/month limit. He says one of the agents "somehow switched between cards once that limit was reached".

Source

Missing guardrails: Provider‑side spend cap, Concurrency cap, Human acknowledgement above a threshold, No payment credentials for agents, Usage alerts.

Source

A local daily budget check you can run today… assumes you log one line per call as date,tokens in a CSV.

Source