How I Built a RAG Search Engine for 10,000 Documents — and Why I Eventually Dropped Answer Generation

Описание и применение

Создана поисковая система на основе RAG для 10 000 религиозных документов, где изначальная плотная векторная индексация приводила к нерелевантным результатам. Добавлен гибридный поиск, фильтрация по косинусному сходству и переоценка с помощью Cohere Rerank, что повысило точность поиска.

Задача

Retrieval in a specialized Russian‑language religious corpus produced semantically similar but context‑irrelevant results, undermining user trust.

Результат

The search engine now reliably returns documents that truly answer user queries, avoiding misleading summaries and maintaining user trust.

Рабочий процесс

• Convert JSON to JSONL with per‑record cleanup
• Chunk each record into title‑question and answer‑footnote
• Generate OpenAI text‑embedding‑3‑large vectors
• Store chunks in Qdrant with deterministic IDs
• For a query: normalize text, embed, retrieve dense and sparse results
• Fuse rankings with Reciprocal Rank Fusion
• Filter candidates by cosine similarity thresholds (0.32/0.20)
• Rerank candidates with Cohere Rerank
• Apply final relevance threshold (0.09)
• Return ranked documents or empty set if none meet threshold

Доказательства

Semantic similarity doesn’t necessarily mean a document answers the question.

Источник

I introduced an additional cosine similarity check using the dense vectors.

Источник

I also introduced another relevance threshold after reranking.

Источник