When the Token Well Runs Dry: A Degradation State Machine for LLM Services
Описание и применение
Создайте машину состояний, которая отслеживает использование токенов и переключает сервис в режимы NORMAL, WARNING, DEGRADED и EXHAUSTED, используя кэш и приоритизацию запросов.
Задача
LLM services can abruptly fail when free token quotas are exhausted, causing 500 errors and user loss.
Результат
The implementation keeps the LLM service alive during quota exhaustion by serving cached responses, prioritizing critical requests, and gracefully degrading to 503 when necessary.
Рабочий процесс
- 1. Track quota usage with a QuotaTracker that polls the usage endpoint and calculates usage ratio.
- 2. Store responses in a ResponseCache keyed by prompt hash to enable quick retrieval.
- 3. Classify incoming requests as critical, cheap, or normal based on payload fields.
- 4. Route each request through a router that checks the current quota state and applies the appropriate behavior: serve from cache, allow critical requests, or return 503 when exhausted.
Доказательства
The state machine is the core.
Cache responses
- Дата записи: 2026-09-07T08:22:36.537Z