One Project Ate the Shared Free Tier: A Per-Project Quota Pattern for LLM Gateways

Описание и применение

Много проектов использовали один LLM‑гейтвей и общий лимит токенов, что привело к отказу всех при исчерпании одного проекта. Внедрённая система квот на уровне проекта с предварительной проверкой, реконсиляцией и очередью предотвратила подобные сбои.

Задача

Multiple projects shared a single LLM gateway, API key, and 10‑million‑token allowance, causing one project’s batch job to exhaust the quota and break all others.

Результат

The per‑project quota system prevented a single project from consuming the shared allowance, added graceful degradation, and ensured accurate usage tracking even after restarts.

Рабочий процесс

• Introduce project ID in request header or path
• Define ProjectQuota objects with daily token and per‑minute limits
• Implement precheck to estimate tokens and reject if over budget
• Forward request to LLM if precheck passes
• After response, reconcile actual token usage and update counters
• Apply degradation ladder: warn at 80%, queue at 100%, reject over per‑minute cap
• Persist usage in SQLite and lazily reset daily counters on first request of new day

Доказательства

Three projects shared one gateway, one API key, and one 10-million-token allowance.

Источник

The pre-check prevented the common failure mode where a retry loop consumed the entire allowance before any monitoring alert could fire.

Источник