PageIndex RAG: Stop Paying Twice for Overlapping Evidence

Описание и применение

Рассматривается проблема дублирования доказательств при использовании PageIndex. Предложен Python‑скрипт, который удаляет повторяющиеся страницы, ограничивает размер доказательств и формирует стабильный префикс запроса, предотвращая двойную оплату и повышая эффективность кэширования.

Задача

Retrieving overlapping document sections causes duplicate evidence to be sent to the answer model, leading to double billing for identical content.

Результат

The provided Python assembler deduplicates overlapping pages, enforces a 24,000-byte evidence budget, and outputs a deterministic prompt prefix. This prevents double billing for identical evidence and enables efficient prompt caching.

Рабочий процесс

• 1. Ingest documents: fingerprint source, preserve page text, build the tree, and version artifacts.
• 2. Retrieve: authorize access, navigate the tree, and resolve selected nodes into source-page references.
• 3. Assemble evidence: deduplicate pages by (document_sha256, page) key, enforce an evidence budget, and construct a stable prompt prefix.
• 4. Cache the stable prefix: store the policy-and-evidence prefix separately from the changing question to enable prompt caching.
• 5. Generate answer: send the assembled prompt to the model, ensuring token limits are respected.

Доказательства

You retrieve three relevant document sections. Two of them cite the same page. Your RAG pipeline concatenates all three and sends 18,000 tokens to the answer model.

Источник

If a parent section and one of its children both reference the same page, you need deduplication before inference—not after the bill arrives.

Источник

The assembler passed local checks for deterministic ordering, duplicate removal, conflicting page text, and budget rejection.

Источник

The 24,000-byte limit is an illustrative transport guard, not a token estimate.

Источник