I Built a Semantic Cache for RAG. The Hard Part Was Knowing When NOT to Cache

Description and use

A semantic cache for RAG that employs an entity guard and model routing reduces costs by up to 94 % and speeds up repeat-heavy workloads, while maintaining answer quality.

Problem

Determining when to reuse cached answers in Retrieval-Augmented Generation (RAG) without returning incorrect responses for similar but distinct queries.

Outcome

Full Weir reduced cost per 1,000 requests from $0.0988 to $0.0786 (20% reduction) in a cold‑pass evaluation and from $0.0970 to $0.0055 (94% reduction) in a repeat‑heavy replay, while keeping judge and key‑fact scores unchanged. Median latency increased modestly in cold‑pass but dropped dramatically (to 14 ms) in repeat‑heavy replay.

Workflow

• 1. Receive user query
• 2. Compute embedding similarity to cached queries
• 3. Apply entity guard to check distinguishing information (numbers, negation, domain terms)
• 4. If guard passes, return cached answer
• 5. If guard fails, retrieve documents and generate answer with chosen model tier
• 6. Log request, cost, and latency for monitoring

Evidence

Full Weir reduced measured cost per 1,000 requests from $0.0988 to $0.0786—a 20% reduction.

Source

On this workload, the measured cost reduction was approximately 94%, and median latency was about 50 times lower.

Source

Adding the entity guard reduced wrong accepted cache hits to zero on that test set.

Source