I've Built RAG Infrastructure Several Times. Last Week Was the First Time I Actually Benchmarked It

说明与用途

作者构建了 RAG 系统但未进行正式评估。通过添加确定性基准、修复 FTS 和构建错误以及过滤停用词，召回率从 0 提升至 0.13，验证了混合搜索的价值。

问题

Retrieval quality in a RAG system was unverified; key channels (FTS, vector, hybrid) were not functioning correctly, leading to zero recall for keyword queries and missing hybrid benefits.

结果

Benchmarking revealed zero recall for FTS, missing hybrid benefits, and two production bugs. After fixing stopword filtering and database issues, FTS recall improved to 0.1314 and hybrid recall to 0.9826, confirming hybrid search value. Embedding choice and reranker impact were quantified, providing a data‑driven roadmap for future improvements.

工作流

• 1. List all retrieval decisions and document rationale. 2. Create deterministic benchmark harness with MIRACL‑zh queries. 3. Run smoke test with mock embeddings to validate harness. 4. Detect production bugs (SQLite FK, missing vector extension) and fix. 5. Execute full benchmark with real embeddings (bge‑m3) across four configurations. 6. Analyze results: FTS recall 0.0000 → 0.1314 after stopword filtering; hybrid 0.9799 → 0.9826. 7. Measure impact of embedding model swap and reranker. 8. Document findings and update README. 9. Publish RETRIEVAL_BENCHMARK.md and make eval binary available.

证据

Real model in (bge-m3), full 200 queries, and the results table came back with a row of 0.0000s. The FTS channel. Zero recall.

来源

The bug sat one column over: hybrid scored 0.9799 — identical to vector-only, digit for digit.

来源

The fix filters the query side: strip punctuation, single-character function words (有, 的, 了), question fillers (哪些, 怎么, 为什么).

来源