A five-minute test for any RAG assistant: two documents with conflicting dates

Description and use

The test checks whether a RAG assistant cites the current document, honestly reports missing facts, and respects access limits, uncovering generation errors invisible to standard metrics.

Problem

RAG systems often answer fluently from wrong evidence or hallucinate missing facts, making failures invisible in standard metrics.

Outcome

The test reveals whether a RAG system correctly cites current evidence, admits missing facts, and respects access permissions, catching generation-side failures invisible to standard metrics.

Workflow

• Create two synthetic documents about the same event with conflicting dates.
• Upload both documents to the RAG system.
• Ask a question whose answer depends on the current document (e.g., "When does the policy change take effect?").
• Verify the answer cites Document B and not Document A.
• Ask a question only answerable from Document B (e.g., "Does it apply to the Basic plan?") and check the tool reports which document was used.
• Ask a question about a fact absent from both documents (e.g., "Who approved the change?") and ensure the system says it is not in the documents.
• Test access boundaries: remove Document B, re-ask the same questions, and confirm the system degrades honestly without silently falling back to Document A.
• Swap the dates in the documents and repeat the test to ensure the system does not rely on file order or names.

Evidence

A regression suite with fixed questions (LLM-as-judge grading, retrieval-vs-generation failure tagging) is worth building — but it has a blind spot: it measures the cases you thought to ask.

Source

The failures this test catches are generation-side failures that look correct — fluent answer, wrong provenance.

Source