How I Built a RAG Search Engine for 10,000 Documents — and Why I Eventually Dropped Answer Generation

Description and use

A RAG‑based search engine for 10,000 religious documents initially returned semantically close but irrelevant results. Hybrid dense‑sparse retrieval, cosine similarity filtering, and Cohere Rerank were added to improve relevance and trust.

Problem

Retrieval in a specialized Russian‑language religious corpus produced semantically similar but context‑irrelevant results, undermining user trust.

Outcome

The search engine now reliably returns documents that truly answer user queries, avoiding misleading summaries and maintaining user trust.

Workflow

• Convert JSON to JSONL with per‑record cleanup
• Chunk each record into title‑question and answer‑footnote
• Generate OpenAI text‑embedding‑3‑large vectors
• Store chunks in Qdrant with deterministic IDs
• For a query: normalize text, embed, retrieve dense and sparse results
• Fuse rankings with Reciprocal Rank Fusion
• Filter candidates by cosine similarity thresholds (0.32/0.20)
• Rerank candidates with Cohere Rerank
• Apply final relevance threshold (0.09)
• Return ranked documents or empty set if none meet threshold

Evidence

Semantic similarity doesn’t necessarily mean a document answers the question.

Source

I introduced an additional cosine similarity check using the dense vectors.

Source

I also introduced another relevance threshold after reranking.

Source