My RAG tool refused to answer a question my notes clearly answered

Description and use

A custom RAG system failed to retrieve answers because chunks were cut mid‑word and short sections were diluted. Soft chunking, an optimal chunk size (~350 chars), and a refusal prompt fixed the issue, enabling correct answers or appropriate refusals.

Problem

A retrieval‑augmented generation (RAG) system built from personal notes failed to retrieve the correct answer because of chunking issues and silent failures.

Outcome

After adjusting chunk boundaries and adding a refusal prompt, the RAG system correctly answers questions found in the notes and appropriately refuses when no answer exists.

Workflow

• 1. Tokenize notes into 500‑character chunks with 100‑character overlap.
• 2. Observe retrieval returns mid‑word fragments, causing no match.
• 3. Increase chunk size to 900 characters, leading to diluted embeddings for short sections.
• 4. Experiment with chunk size 350 and overlap 70, matching whole sections.
• 5. Add a soft cut function to avoid mid‑word splits.
• 6. Test with real and unanswerable questions, noting similarity scores.
• 7. Replace hard relevance floor with a prompt that allows the model to refuse when no answer is found.
• 8. Deploy updated pipeline and verify correct behavior.

Evidence

I built a retrieval-augmented generation tool over my own notes — no LangChain, no LlamaIndex, every stage by hand.

Source

I can't find the answer to that in your notes.

Source

The answer was there. I could point at the line.

Source

My 500-character cut had landed mid-word. Half the sentence in chunk N, half in chunk N+1, and neither half complete enough to match anything.

Source

The fix is to stop treating chunk size as a hard boundary — scan backwards for somewhere sensible to cut.

Source

At 0.35 the bad question is silenced and two real ones drop to two passages. At 0.40 they return nothing.

Source

What works instead is the prompt — giving the model an acceptable way out: 3. If the passages do not contain the answer, say so plainly...

Source