PageIndex RAG: Stop Paying Twice for Overlapping Evidence

Description and use

The article addresses duplicate evidence in PageIndex RAG pipelines. A Python script deduplicates pages, enforces a byte budget, and builds a stable prompt prefix, preventing double billing and enabling efficient prompt caching.

Problem

Retrieving overlapping document sections causes duplicate evidence to be sent to the answer model, leading to double billing for identical content.

Outcome

The provided Python assembler deduplicates overlapping pages, enforces a 24,000-byte evidence budget, and outputs a deterministic prompt prefix. This prevents double billing for identical evidence and enables efficient prompt caching.

Workflow

• 1. Ingest documents: fingerprint source, preserve page text, build the tree, and version artifacts.
• 2. Retrieve: authorize access, navigate the tree, and resolve selected nodes into source-page references.
• 3. Assemble evidence: deduplicate pages by (document_sha256, page) key, enforce an evidence budget, and construct a stable prompt prefix.
• 4. Cache the stable prefix: store the policy-and-evidence prefix separately from the changing question to enable prompt caching.
• 5. Generate answer: send the assembled prompt to the model, ensuring token limits are respected.

Evidence

You retrieve three relevant document sections. Two of them cite the same page. Your RAG pipeline concatenates all three and sends 18,000 tokens to the answer model.

Source

If a parent section and one of its children both reference the same page, you need deduplication before inference—not after the bill arrives.

Source

The assembler passed local checks for deterministic ordering, duplicate removal, conflicting page text, and budget rejection.

Source

The 24,000-byte limit is an illustrative transport guard, not a token estimate.

Source