Building a Voice RAG System Under a 200ms Latency Budget

说明与用途

构建了一个语音驱动的 RAG 系统，在 200 ms 延迟预算内实现 100% 请求成功率，采用混合检索、RRF、MMR 和证据验证。

问题

Design a voice‑driven Retrieval‑Augmented Generation (RAG) system that delivers grounded answers within a strict 200 ms latency budget while refusing to answer when evidence is insufficient.

结果

The pipeline processed 927 benchmark runs, achieving P50 = 7.87 ms, P70 = 9.62 ms, P100 = 40.71 ms, and 100 % of requests within the 200 ms budget.

工作流

• Capture voice input
• Convert to text via Speech‑to‑Text
• Guard query for safety
• Retrieve candidates using dense and BM25+ retrieval
• Fuse results with RRF
• Rerank with MMR
• Check retrieval confidence
• Generate grounded answer with Claude
• Verify grounding and prompt safety
• Return answer or refusal

证据

The final results: P50 = 7.87 ms, P70 = 9.62 ms, P100 = 40.71 ms, Within 200 ms = 100 %

来源

Built with Python · FastAPI · BM25+ · Dense Retrieval · RRF · MMR · Sarvam · Claude · Docker

来源