I Built a Semantic Cache for RAG. The Hard Part Was Knowing When NOT to Cache

Описание и применение

Создан семантический кэш для RAG, который использует защиту по сущностям и маршрутизацию моделей, снижая затраты до 94 % и ускоряя ответы в повторяющихся нагрузках, но требует осторожности при обработке похожих запросов.

Задача

Determining when to reuse cached answers in Retrieval-Augmented Generation (RAG) without returning incorrect responses for similar but distinct queries.

Результат

Full Weir reduced cost per 1,000 requests from $0.0988 to $0.0786 (20% reduction) in a cold‑pass evaluation and from $0.0970 to $0.0055 (94% reduction) in a repeat‑heavy replay, while keeping judge and key‑fact scores unchanged. Median latency increased modestly in cold‑pass but dropped dramatically (to 14 ms) in repeat‑heavy replay.

Рабочий процесс

• 1. Receive user query
• 2. Compute embedding similarity to cached queries
• 3. Apply entity guard to check distinguishing information (numbers, negation, domain terms)
• 4. If guard passes, return cached answer
• 5. If guard fails, retrieve documents and generate answer with chosen model tier
• 6. Log request, cost, and latency for monitoring

Доказательства

Full Weir reduced measured cost per 1,000 requests from $0.0988 to $0.0786—a 20% reduction.

Источник

On this workload, the measured cost reduction was approximately 94%, and median latency was about 50 times lower.

Источник

Adding the entity guard reduced wrong accepted cache hits to zero on that test set.

Источник