When the Token Well Runs Dry: A Degradation State Machine for LLM Services

说明与用途

构建一个状态机，监控令牌使用情况，并通过 NORMAL、WARNING、DEGRADED 和 EXHAUSTED 模式切换服务，利用缓存和请求优先级。

问题

LLM services can abruptly fail when free token quotas are exhausted, causing 500 errors and user loss.

结果

The implementation keeps the LLM service alive during quota exhaustion by serving cached responses, prioritizing critical requests, and gracefully degrading to 503 when necessary.

工作流

• 1. Track quota usage with a QuotaTracker that polls the usage endpoint and calculates usage ratio.
• 2. Store responses in a ResponseCache keyed by prompt hash to enable quick retrieval.
• 3. Classify incoming requests as critical, cheap, or normal based on payload fields.
• 4. Route each request through a router that checks the current quota state and applies the appropriate behavior: serve from cache, allow critical requests, or return 503 when exhausted.

证据

The state machine is the core.

来源

Cache responses

来源