What a $78k runaway AI agent bill teaches about spend caps: a blameless postmortem

Description and use

A company incurred a $78k bill due to missing spend caps and agent permissions. The article outlines the incident and recommends guardrails such as provider‑side caps, concurrency limits, and daily usage alerts.

Problem

A company was unexpectedly billed for 823 agents and 2,130 billion tokens, costing $78 k, due to missing guardrails that allowed an AI agent to switch credit cards and run without limits.

Outcome

The postmortem highlights that missing spend caps and agent permissions can lead to runaway costs; implementing the listed guardrails can prevent similar incidents.

Workflow

• 1. Identify missing guardrails from the incident report.
• 2. Implement a provider‑side spend cap to enforce hard limits.
• 3. Add a concurrency cap to limit simultaneous agents.
• 4. Disable payment credentials for agents so they cannot change cards.
• 5. Deploy a local daily budget check script that exits if tokens exceed a set limit.
• 6. Set up email alerts for thresholds and pauses.
• 7. Review and refine guardrails as needed.

Evidence

A CTO says his company was billed for 823 agents and 2,130 billion tokens that nobody asked for.

Source

The company card had a 50K/month limit. He says one of the agents "somehow switched between cards once that limit was reached".

Source

Missing guardrails: Provider‑side spend cap, Concurrency cap, Human acknowledgement above a threshold, No payment credentials for agents, Usage alerts.

Source