Postmortem: The Tuesday Our AI Bill Had No Owner

说明与用途

缺少 Vertex AI 请求标签导致夜间工单摘要工作流中的重试循环，AI 成本翻三倍。实施标签、标签体系和强制执行将调查时间缩短为一次查询，并防止未来成本激增。

问题

Vertex AI spend tripled overnight because model calls lacked request labels, preventing per-call attribution and real-time cost visibility.

结果

The investigation revealed that missing request labels caused a retry loop in a nightly summarization workflow, tripling AI spend. Adding labels and enforcing them reduced investigation time to a single query and prevented future cost spikes.

工作流

• Deploy prompt_version=v4 for ticket summarization workflow
• Nightly job starts and reviewer rejects drafts, triggering rewrites
• Daily cost alert fires at 07:40
• SRE filters Cloud Billing by project and sees aggregated cost
• Slack thread identifies no AI-related shipping but a prompt update
• SRE greps logs for model call volume
• Developer notes job ran six hours instead of forty minutes
• Root cause confirmed: review loop with growing context
• Rollback to v3, lower retry cap, rerun job successfully

证据

None of our Vertex AI calls carried request labels.

来源

The billing console told us exactly how much we spent. It could not tell us who, which feature, or which step.

来源

The review step in the summarization workflow rejected most drafts, causing repeated rewrites with growing context.

来源

Three labels would have turned our two-hour investigation into one query.

来源