Postmortem: The Tuesday Our AI Bill Had No Owner

Description and use

Missing request labels on Vertex AI calls caused a retry loop in a nightly ticket summarization workflow, tripling AI spend. Implementing labels, a taxonomy, and enforcement reduced investigation time to a single query and prevented future cost spikes.

Problem

Vertex AI spend tripled overnight because model calls lacked request labels, preventing per-call attribution and real-time cost visibility.

Outcome

The investigation revealed that missing request labels caused a retry loop in a nightly summarization workflow, tripling AI spend. Adding labels and enforcing them reduced investigation time to a single query and prevented future cost spikes.

Workflow

• Deploy prompt_version=v4 for ticket summarization workflow
• Nightly job starts and reviewer rejects drafts, triggering rewrites
• Daily cost alert fires at 07:40
• SRE filters Cloud Billing by project and sees aggregated cost
• Slack thread identifies no AI-related shipping but a prompt update
• SRE greps logs for model call volume
• Developer notes job ran six hours instead of forty minutes
• Root cause confirmed: review loop with growing context
• Rollback to v3, lower retry cap, rerun job successfully

Evidence

None of our Vertex AI calls carried request labels.

Source

The billing console told us exactly how much we spent. It could not tell us who, which feature, or which step.

Source

The review step in the summarization workflow rejected most drafts, causing repeated rewrites with growing context.

Source

Three labels would have turned our two-hour investigation into one query.

Source