Cap LLM Spend per Team with Virtual Keys

Description and use

Implemented LLM cost tracking, attribution, and budget enforcement at the gateway level.

Problem

Organizations need to know who is spending on LLMs and prevent runaway costs.

Outcome

Implemented per‑request cost tracking, attribution, and budget enforcement at the gateway level, enabling real‑time spend visibility and automated request blocking.

Workflow

• Load a JSON price catalog into a ConfigMap and reference it via EnterpriseAgentgatewayParameters.
• Create virtual keys as Secrets containing user/team metadata.
• Configure the gateway to use Strict API key authentication and enable entBudgetEnforcement.
• Define budgets in an EnterpriseAgentgatewayBudget resource with token and USD limits.
• Run the demo with ./demo.sh --act 8 to observe cost attribution and budget enforcement.
• Verify budgets via kubectl get eagbud and monitor logs for llm.cost fields.

Evidence

The gateway picks it up through modelCatalog.sources on the EnterpriseAgentgatewayParameters it already references

Source

A virtual key is a gateway‑issued token carrying attribution metadata

Source

Strict means a valid key is required, so on this route unattributed spend cannot happen at all rather than merely being visible

Source