Building a Voice RAG System Under a 200ms Latency Budget

Description and use

A voice‑driven RAG system achieving 100 % of requests under a 200 ms latency budget, using hybrid retrieval, RRF, MMR, and evidence verification.

Problem

Design a voice‑driven Retrieval‑Augmented Generation (RAG) system that delivers grounded answers within a strict 200 ms latency budget while refusing to answer when evidence is insufficient.

Outcome

The pipeline processed 927 benchmark runs, achieving P50 = 7.87 ms, P70 = 9.62 ms, P100 = 40.71 ms, and 100 % of requests within the 200 ms budget.

Workflow

• Capture voice input
• Convert to text via Speech‑to‑Text
• Guard query for safety
• Retrieve candidates using dense and BM25+ retrieval
• Fuse results with RRF
• Rerank with MMR
• Check retrieval confidence
• Generate grounded answer with Claude
• Verify grounding and prompt safety
• Return answer or refusal

Evidence

The final results: P50 = 7.87 ms, P70 = 9.62 ms, P100 = 40.71 ms, Within 200 ms = 100 %

Source

Built with Python · FastAPI · BM25+ · Dense Retrieval · RRF · MMR · Sarvam · Claude · Docker

Source