When the Token Well Runs Dry: A Degradation State Machine for LLM Services

Description and use

Build a state machine that monitors token usage and switches the service through NORMAL, WARNING, DEGRADED, and EXHAUSTED modes, leveraging caching and request prioritization.

Problem

LLM services can abruptly fail when free token quotas are exhausted, causing 500 errors and user loss.

Outcome

The implementation keeps the LLM service alive during quota exhaustion by serving cached responses, prioritizing critical requests, and gracefully degrading to 503 when necessary.

Workflow

• 1. Track quota usage with a QuotaTracker that polls the usage endpoint and calculates usage ratio.
• 2. Store responses in a ResponseCache keyed by prompt hash to enable quick retrieval.
• 3. Classify incoming requests as critical, cheap, or normal based on payload fields.
• 4. Route each request through a router that checks the current quota state and applies the appropriate behavior: serve from cache, allow critical requests, or return 503 when exhausted.

Evidence

The state machine is the core.

Source

Cache responses

Source