Reranking Models: The Cheapest RAG Quality Upgrade

Description and use

Adding a cross‑encoder reranker between dense vector retrieval and generation boosts RAG answer accuracy from 61 % to 84 % at a cost of under $40 per month.

Problem

RAG systems often return irrelevant chunks because bi‑encoder retrieval misses fine‑grained relevance signals, leading to low answer accuracy.

Outcome

Adding a cheap cross‑encoder reranker between retrieval and generation improved answer accuracy from 61 % to 84 % and cost less than $40 per month.

Workflow

• 1. Run bi‑encoder dense vector search to retrieve top 50‑100 candidates. 2. Merge and dedupe the candidate list. 3. Apply a cross‑encoder reranker (e.g., BAAI/bge‑reranker‑v2‑m3) to score each candidate against the query. 4. Keep the top 5‑10 ranked chunks. 5. Pass those chunks to the LLM prompt for generation.

Evidence

Answer accuracy on our test set went from 61% to 84%

Source

The reranker cost less than $40 that month.

Source

Hybrid retrieval alone moved recall of the correct chunk into the top 50 from 78% to 91%.

Source

Take 10,000 queries per month with 50 candidates reranked per query. With Cohere, each query is one search call since 50 docs fit the 100-doc limit, so 10,000 searches cost $20 per month.

Source