Free Model Tokens Are a Hard Budget. Enforce Them in Code, Not in Discipline

说明与用途

自动代理，预留并跟踪模型令牌，强制执行免费令牌预算，防止过度消耗。

问题

Free model token allowances can be exhausted silently by runaway or retry loops, leading to unexpected service outages and wasted resources.

结果

The proxy enforces a hard token budget, preventing accidental overuse of free allowances and providing clear error handling when limits are reached.

工作流

• 1. Initialize SQLite DB with a usage table.
• 2. On each request, estimate input tokens (len(body)//4).
• 3. Acquire an asyncio lock to serialize budget checks.
• 4. If spent_today + estimate >= DAILY_BUDGET, return 429 with error.
• 5. Insert a reserved record into the DB.
• 6. Forward request to upstream model API via httpx.
• 7. Record actual tokens used from the response and update the DB record.
• 8. Return upstream response to the client.

证据

I built a small budget proxy to prove the point, and the first version failed in a predictable way.

来源

The fix is the same pattern you use for inventory or seats: reserve first, settle later.

来源