Free Model Tokens Are a Hard Budget. Enforce Them in Code, Not in Discipline

Описание и применение

Автоматический прокси, который резервирует и отслеживает токены модели, предотвращая их избыточное потребление в рамках бесплатного лимита.

Задача

Free model token allowances can be exhausted silently by runaway or retry loops, leading to unexpected service outages and wasted resources.

Результат

The proxy enforces a hard token budget, preventing accidental overuse of free allowances and providing clear error handling when limits are reached.

Рабочий процесс

• 1. Initialize SQLite DB with a usage table.
• 2. On each request, estimate input tokens (len(body)//4).
• 3. Acquire an asyncio lock to serialize budget checks.
• 4. If spent_today + estimate >= DAILY_BUDGET, return 429 with error.
• 5. Insert a reserved record into the DB.
• 6. Forward request to upstream model API via httpx.
• 7. Record actual tokens used from the response and update the DB record.
• 8. Return upstream response to the client.

Доказательства

I built a small budget proxy to prove the point, and the first version failed in a predictable way.

Источник

The fix is the same pattern you use for inventory or seats: reserve first, settle later.

Источник