A five-minute test for any RAG assistant: two documents with conflicting dates

Описание и применение

Тест проверяет, ссылается ли RAG‑ассистент на актуальный документ, честно сообщает о недостающих фактах и соблюдает ограничения доступа, выявляя ошибки генерации, которые не видны в стандартных метриках.

Задача

RAG systems often answer fluently from wrong evidence or hallucinate missing facts, making failures invisible in standard metrics.

Результат

The test reveals whether a RAG system correctly cites current evidence, admits missing facts, and respects access permissions, catching generation-side failures invisible to standard metrics.

Рабочий процесс

• Create two synthetic documents about the same event with conflicting dates.
• Upload both documents to the RAG system.
• Ask a question whose answer depends on the current document (e.g., "When does the policy change take effect?").
• Verify the answer cites Document B and not Document A.
• Ask a question only answerable from Document B (e.g., "Does it apply to the Basic plan?") and check the tool reports which document was used.
• Ask a question about a fact absent from both documents (e.g., "Who approved the change?") and ensure the system says it is not in the documents.
• Test access boundaries: remove Document B, re-ask the same questions, and confirm the system degrades honestly without silently falling back to Document A.
• Swap the dates in the documents and repeat the test to ensure the system does not rely on file order or names.

Доказательства

A regression suite with fixed questions (LLM-as-judge grading, retrieval-vs-generation failure tagging) is worth building — but it has a blind spot: it measures the cases you thought to ask.

Источник

The failures this test catches are generation-side failures that look correct — fluent answer, wrong provenance.

Источник