I've Built RAG Infrastructure Several Times. Last Week Was the First Time I Actually Benchmarked It

Description and use

The author built a RAG system without formal metrics. By adding a deterministic benchmark, fixing FTS and build bugs, and filtering stopwords, recall improved from 0 to 0.13 and hybrid search proved beneficial.

Problem

Retrieval quality in a RAG system was unverified; key channels (FTS, vector, hybrid) were not functioning correctly, leading to zero recall for keyword queries and missing hybrid benefits.

Outcome

Benchmarking revealed zero recall for FTS, missing hybrid benefits, and two production bugs. After fixing stopword filtering and database issues, FTS recall improved to 0.1314 and hybrid recall to 0.9826, confirming hybrid search value. Embedding choice and reranker impact were quantified, providing a data‑driven roadmap for future improvements.

Workflow

• 1. List all retrieval decisions and document rationale. 2. Create deterministic benchmark harness with MIRACL‑zh queries. 3. Run smoke test with mock embeddings to validate harness. 4. Detect production bugs (SQLite FK, missing vector extension) and fix. 5. Execute full benchmark with real embeddings (bge‑m3) across four configurations. 6. Analyze results: FTS recall 0.0000 → 0.1314 after stopword filtering; hybrid 0.9799 → 0.9826. 7. Measure impact of embedding model swap and reranker. 8. Document findings and update README. 9. Publish RETRIEVAL_BENCHMARK.md and make eval binary available.

Evidence

Real model in (bge-m3), full 200 queries, and the results table came back with a row of 0.0000s. The FTS channel. Zero recall.

Source

The bug sat one column over: hybrid scored 0.9799 — identical to vector-only, digit for digit.

Source

The fix filters the query side: strip punctuation, single-character function words (有, 的, 了), question fillers (哪些, 怎么, 为什么).

Source