We Tested Our Own AI Safety Checks. They Caught Nothing.
说明与用途
一款使用来源检查的罗马尼亚问答演示无法检测到12个中6个虚构答案,表明仅凭来源检查无法防止看似可信但不正确的回答。
问题
A Romanian question‑answering demo that refuses any answer it cannot trace to a source document failed to detect invented answers that were built from true parts of the source.
结果
The model alone invented answers in 4 of 12 cases where evidence was removed; behind gates B, C and D it invented 6 of 12. All 24 invented answers passed the gates, showing that provenance checks alone cannot detect plausible but incorrect answers.
工作流
- 1. Build 12 question pairs: one with the answer paragraph and one with the paragraph removed.
- 2. Run the model alone on all 24 items.
- 3. Run the model behind gates B, C and D on all 24 items.
- 4. Record counts of correct answers, “not in text” responses, and invented answers.
- 5. Analyze why invented answers passed the gates.
- 6. Publish results and source code for reproducibility.
证据
The model invented an answer in 4 of 12 cases where the evidence had been removed.
Gates B, C and D refused 0 of 24 items.
The gates check provenance: whether the words came from the document. They do not check relevance: whether the sentence answers the question that was asked.
The gated mode also uses the demo's own prompt, which is worded differently from the bare prompt.
- 记录日期: 2026-10-07T14:55:57.745Z