A five-minute test for any RAG assistant: two documents with conflicting dates

说明与用途

该测试检查 RAG 助手是否引用当前文档、诚实报告缺失事实并遵守访问限制，揭示标准指标无法检测的生成错误。

问题

RAG systems often answer fluently from wrong evidence or hallucinate missing facts, making failures invisible in standard metrics.

结果

The test reveals whether a RAG system correctly cites current evidence, admits missing facts, and respects access permissions, catching generation-side failures invisible to standard metrics.

工作流

• Create two synthetic documents about the same event with conflicting dates.
• Upload both documents to the RAG system.
• Ask a question whose answer depends on the current document (e.g., "When does the policy change take effect?").
• Verify the answer cites Document B and not Document A.
• Ask a question only answerable from Document B (e.g., "Does it apply to the Basic plan?") and check the tool reports which document was used.
• Ask a question about a fact absent from both documents (e.g., "Who approved the change?") and ensure the system says it is not in the documents.
• Test access boundaries: remove Document B, re-ask the same questions, and confirm the system degrades honestly without silently falling back to Document A.
• Swap the dates in the documents and repeat the test to ensure the system does not rely on file order or names.

证据

A regression suite with fixed questions (LLM-as-judge grading, retrieval-vs-generation failure tagging) is worth building — but it has a blind spot: it measures the cases you thought to ask.

来源

The failures this test catches are generation-side failures that look correct — fluent answer, wrong provenance.

来源