Slashing LLM Bills by 58%: Building an OpenTelemetry Semantic Cache Proxy in FastAPI

Описание и применение

Внедрение FastAPI‑прокси с Redis‑векторным кэшем и OpenTelemetry сократило расходы на LLM на 58 % за счёт кэширования семантически схожих запросов и предоставления подробной наблюдаемости.

Задача

Enterprise SaaS applications spend 30‑60% of LLM queries on duplicate or near‑duplicate prompts, inflating inference bills and causing unpredictable latency.

Результат

The proxy achieved a 58% reduction in LLM spending by serving cached completions for semantically similar prompts and providing detailed observability.

Рабочий процесс

• Intercept request and extract tenant headers
• Validate tenant token budget
• Generate prompt embedding with lightweight model
• Query Redis vector index for nearest neighbor (cosine ≥ 0.92)
• Return cached completion if hit
• If miss, check token budget and forward to upstream provider
• Emit OpenTelemetry spans with prompt/completion tokens, latency, and cache status

Доказательства

between 30% and 60% of LLM queries in enterprise SaaS applications are duplicates or slight paraphrases

Источник

Hashing prompts (SHA256(prompt)) in Redis yields miserable cache hit rates (under 4%)

Источник

Optimized Self-Hosted Flow: Client ──> FastAPI Proxy ──> Redis Vector Search (Cosine >= 0.92) ──> Cache HIT (0ms upstream cost)

Источник