Slashing LLM Bills by 58%: Building an OpenTelemetry Semantic Cache Proxy in FastAPI

Description and use

Deploying a FastAPI reverse proxy with Redis vector cache and OpenTelemetry cut LLM costs by 58 % by caching semantically similar prompts and delivering detailed observability.

Problem

Enterprise SaaS applications spend 30‑60% of LLM queries on duplicate or near‑duplicate prompts, inflating inference bills and causing unpredictable latency.

Outcome

The proxy achieved a 58% reduction in LLM spending by serving cached completions for semantically similar prompts and providing detailed observability.

Workflow

• Intercept request and extract tenant headers
• Validate tenant token budget
• Generate prompt embedding with lightweight model
• Query Redis vector index for nearest neighbor (cosine ≥ 0.92)
• Return cached completion if hit
• If miss, check token budget and forward to upstream provider
• Emit OpenTelemetry spans with prompt/completion tokens, latency, and cache status

Evidence

between 30% and 60% of LLM queries in enterprise SaaS applications are duplicates or slight paraphrases

Source

Hashing prompts (SHA256(prompt)) in Redis yields miserable cache hit rates (under 4%)

Source

Optimized Self-Hosted Flow: Client ──> FastAPI Proxy ──> Redis Vector Search (Cosine >= 0.92) ──> Cache HIT (0ms upstream cost)

Source