Cap LLM Spend per Team with Virtual Keys

Описание и применение

Внедрена система отслеживания стоимости запросов к LLM, атрибуции пользователей/команд и автоматического ограничения расходов через шлюз.

Задача

Organizations need to know who is spending on LLMs and prevent runaway costs.

Результат

Implemented per‑request cost tracking, attribution, and budget enforcement at the gateway level, enabling real‑time spend visibility and automated request blocking.

Рабочий процесс

• Load a JSON price catalog into a ConfigMap and reference it via EnterpriseAgentgatewayParameters.
• Create virtual keys as Secrets containing user/team metadata.
• Configure the gateway to use Strict API key authentication and enable entBudgetEnforcement.
• Define budgets in an EnterpriseAgentgatewayBudget resource with token and USD limits.
• Run the demo with ./demo.sh --act 8 to observe cost attribution and budget enforcement.
• Verify budgets via kubectl get eagbud and monitor logs for llm.cost fields.

Доказательства

The gateway picks it up through modelCatalog.sources on the EnterpriseAgentgatewayParameters it already references

Источник

A virtual key is a gateway‑issued token carrying attribution metadata

Источник

Strict means a valid key is required, so on this route unattributed spend cannot happen at all rather than merely being visible

Источник