Building a Voice RAG System Under a 200ms Latency Budget

Описание и применение

Создан голосовой RAG‑система с 200 мс бюджетом задержки, использующая гибридный поиск, RRF, MMR и проверку доказательств, достигающая 100 % запросов за пределами лимита.

Задача

Design a voice‑driven Retrieval‑Augmented Generation (RAG) system that delivers grounded answers within a strict 200 ms latency budget while refusing to answer when evidence is insufficient.

Результат

The pipeline processed 927 benchmark runs, achieving P50 = 7.87 ms, P70 = 9.62 ms, P100 = 40.71 ms, and 100 % of requests within the 200 ms budget.

Рабочий процесс

• Capture voice input
• Convert to text via Speech‑to‑Text
• Guard query for safety
• Retrieve candidates using dense and BM25+ retrieval
• Fuse results with RRF
• Rerank with MMR
• Check retrieval confidence
• Generate grounded answer with Claude
• Verify grounding and prompt safety
• Return answer or refusal

Доказательства

The final results: P50 = 7.87 ms, P70 = 9.62 ms, P100 = 40.71 ms, Within 200 ms = 100 %

Источник

Built with Python · FastAPI · BM25+ · Dense Retrieval · RRF · MMR · Sarvam · Claude · Docker

Источник