RAG Chunking Strategies That Actually Move Quality

Description and use

Document chunking strategies in RAG systems greatly affect retrieval quality. Leveraging document structure, headings, overlap, metadata, and parent-child retrieval improves answer accuracy.

Problem

Retrieval quality in RAG systems suffers when documents are chunked with default settings, leading to orphan chunks, shredded context, and diluted embeddings.

Outcome

Adopting structure-aware chunking with heading prefixes, overlap, and metadata, plus parent-child retrieval, reduced orphan chunks and improved retrieval relevance, leading to higher-quality answers in RAG systems.

Workflow

• 1. Analyze existing chunking strategy and identify failure modes (orphan, shredded, diluted).
• 2. Choose a chunking strategy based on document type (e.g., structure-aware for knowledge bases).
• 3. Apply heading prefixes to each chunk to embed context.
• 4. Add 10–15% sentence-level overlap between chunks.
• 5. Store metadata (title, section, URL, date) with each chunk.
• 6. Optionally implement parent-child retrieval: index small child chunks (128–256 tokens) pointing to larger parents (512–1,024 tokens).
• 7. Deduplicate parents before prompt assembly and cap total parent tokens per query.
• 8. Evaluate retrieval performance and iterate on chunk size (200–400 tokens for FAQ, 400–700 for docs).

Evidence

Someone split the documents with a default setting on day one, shipped it, and never looked at the chunks again.

Source

Chunking is the highest-impact knob in a RAG system, and it is the one most teams touch exactly once.

Source

This one change fixed a client system where 40 percent of failed lookups were orphan chunks.

Source

Parent-child adds moving parts: a document store alongside the vector store, and join logic in retrieval.

Source