Why Your RAG Chatbot Gives Wrong Answers (And How to Fix It)

Description and use

RAG chatbots frequently return wrong answers due to retrieval issues. Adjusting chunk size, adding overlap, cleaning extracted text, and OCR for scanned PDFs fix the problem.

Problem

RAG chatbots often return incorrect or fabricated answers because the retrieval step fails to provide the correct document excerpts.

Outcome

By tuning chunk size, adding overlap, cleaning extracted text, and handling scanned PDFs, the RAG chatbot’s retrieval accuracy improved, leading to more correct answers.

Workflow

• 1. Print the retrieved chunks to inspect their content.
• 2. Adjust chunk size to 300‑500 words and test multiple sizes.
• 3. Add overlap of ~50 words between chunks to avoid sentence splits.
• 4. Extract text from PDF using pdfplumber and inspect for noise.
• 5. Remove repeated header/footer lines using a frequency counter.
• 6. Detect empty or short pages and flag potential scanned PDFs.
• 7. If scanned, apply OCR (e.g., pytesseract) before extraction.
• 8. Rebuild the index with cleaned chunks and retest queries.

Evidence

If the answer is not in those pieces, fix retrieval (problems 1 to 4 and 7 below).

Source

Fix: start with about 300 to 500 words per chunk.

Source

let each chunk repeat the last 50 words or so of the one before it.

Source

If the text you extract is bad, everything after it is bad too.

Source