Don’t Let a Free LLM Tier Vanish: A Self‑Healing Token Budget in 80 Lines

Описание и применение

Легкая 80‑строчная Python‑служба отслеживает токен‑использование бесплатного LLM и автоматически переключается на резервный провайдер, предотвращая ошибки 429.

Задача

Free LLM tiers run out unexpectedly because code does not track remaining token budget, leading to 429 errors after batch jobs burn through the allowance.

Результат

The 80‑line solution prevents sudden 429 errors by monitoring token usage and automatically failing over to a backup provider before the free tier is exhausted.

Рабочий процесс

• 1. Define a BudgetMonitor class that reads/writes token usage to a CSV file.
• 2. Implement a record method to log each prompt and tokens used.
• 3. Add an under_budget check with a safety margin (default 1000 tokens).
• 4. Create a ProviderRouter that holds provider functions and an active flag.
• 5. In generate(), check monitor.under_budget() before calling the free provider; if false, switch active to backup.
• 6. After each call, record usage via monitor.record().
• 7. Deploy the service on a free server; schedule a cron job to reset or archive usage if needed.

Доказательства

The router keeps a list of provider functions. Each function has the same signature: (prompt, max_tokens) -> (text, tokens_used).

Источник

The safety margin prevents edge‑of‑the‑cliff failures. When you have less than 1,000 tokens left, the router should stop sending new work to the free provider.

Источник