Free Model Tokens Are a Hard Budget. Enforce Them in Code, Not in Discipline

Description and use

An automated proxy that reserves and tracks model tokens to enforce a hard free‑token budget, preventing runaway consumption.

Problem

Free model token allowances can be exhausted silently by runaway or retry loops, leading to unexpected service outages and wasted resources.

Outcome

The proxy enforces a hard token budget, preventing accidental overuse of free allowances and providing clear error handling when limits are reached.

Workflow

• 1. Initialize SQLite DB with a usage table.
• 2. On each request, estimate input tokens (len(body)//4).
• 3. Acquire an asyncio lock to serialize budget checks.
• 4. If spent_today + estimate >= DAILY_BUDGET, return 429 with error.
• 5. Insert a reserved record into the DB.
• 6. Forward request to upstream model API via httpx.
• 7. Record actual tokens used from the response and update the DB record.
• 8. Return upstream response to the client.

Evidence

I built a small budget proxy to prove the point, and the first version failed in a predictable way.

Source

The fix is the same pattern you use for inventory or seats: reserve first, settle later.

Source