I Built a Semantic Cache for RAG. The Hard Part Was Knowing When NOT to Cache

说明与用途

为 RAG 构建了一个语义缓存，使用实体保护和模型路由，在重复请求中将成本降低至 94% 并显著降低延迟，同时保持答案质量。

问题

Determining when to reuse cached answers in Retrieval-Augmented Generation (RAG) without returning incorrect responses for similar but distinct queries.

结果

Full Weir reduced cost per 1,000 requests from $0.0988 to $0.0786 (20% reduction) in a cold‑pass evaluation and from $0.0970 to $0.0055 (94% reduction) in a repeat‑heavy replay, while keeping judge and key‑fact scores unchanged. Median latency increased modestly in cold‑pass but dropped dramatically (to 14 ms) in repeat‑heavy replay.

工作流

• 1. Receive user query
• 2. Compute embedding similarity to cached queries
• 3. Apply entity guard to check distinguishing information (numbers, negation, domain terms)
• 4. If guard passes, return cached answer
• 5. If guard fails, retrieve documents and generate answer with chosen model tier
• 6. Log request, cost, and latency for monitoring

证据

Full Weir reduced measured cost per 1,000 requests from $0.0988 to $0.0786—a 20% reduction.

来源

On this workload, the measured cost reduction was approximately 94%, and median latency was about 50 times lower.

来源

Adding the entity guard reduced wrong accepted cache hits to zero on that test set.

来源