Slashing LLM Bills by 58%: Building an OpenTelemetry Semantic Cache Proxy in FastAPI

说明与用途

使用 FastAPI 反向代理、Redis 向量缓存和 OpenTelemetry，按语义相似的提示进行缓存，降低 LLM 成本 58%，并提供详细可观测性。

问题

Enterprise SaaS applications spend 30‑60% of LLM queries on duplicate or near‑duplicate prompts, inflating inference bills and causing unpredictable latency.

结果

The proxy achieved a 58% reduction in LLM spending by serving cached completions for semantically similar prompts and providing detailed observability.

工作流

• Intercept request and extract tenant headers
• Validate tenant token budget
• Generate prompt embedding with lightweight model
• Query Redis vector index for nearest neighbor (cosine ≥ 0.92)
• Return cached completion if hit
• If miss, check token budget and forward to upstream provider
• Emit OpenTelemetry spans with prompt/completion tokens, latency, and cache status

证据

between 30% and 60% of LLM queries in enterprise SaaS applications are duplicates or slight paraphrases

来源

Hashing prompts (SHA256(prompt)) in Redis yields miserable cache hit rates (under 4%)

来源

Optimized Self-Hosted Flow: Client ──> FastAPI Proxy ──> Redis Vector Search (Cosine >= 0.92) ──> Cache HIT (0ms upstream cost)

来源