Postmortem: The Tuesday Our AI Bill Had No Owner

Описание и применение

В результате отсутствия меток запроса в вызовах Vertex AI расходы на AI выросли в три раза, поскольку цикл повторных попыток в ночном процессе суммирования билетов увеличивал контекст. Добавление меток и их обязательное применение сократило расследование до одного запроса и предотвратило будущие всплески расходов.

Задача

Vertex AI spend tripled overnight because model calls lacked request labels, preventing per-call attribution and real-time cost visibility.

Результат

The investigation revealed that missing request labels caused a retry loop in a nightly summarization workflow, tripling AI spend. Adding labels and enforcing them reduced investigation time to a single query and prevented future cost spikes.

Рабочий процесс

• Deploy prompt_version=v4 for ticket summarization workflow
• Nightly job starts and reviewer rejects drafts, triggering rewrites
• Daily cost alert fires at 07:40
• SRE filters Cloud Billing by project and sees aggregated cost
• Slack thread identifies no AI-related shipping but a prompt update
• SRE greps logs for model call volume
• Developer notes job ran six hours instead of forty minutes
• Root cause confirmed: review loop with growing context
• Rollback to v3, lower retry cap, rerun job successfully

Доказательства

None of our Vertex AI calls carried request labels.

Источник

The billing console told us exactly how much we spent. It could not tell us who, which feature, or which step.

Источник

The review step in the summarization workflow rejected most drafts, causing repeated rewrites with growing context.

Источник

Three labels would have turned our two-hour investigation into one query.

Источник