Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

We Tested Our Own AI Safety Checks. They Caught Nothing.

说明与用途

一款使用来源检查的罗马尼亚问答演示无法检测到12个中6个虚构答案,表明仅凭来源检查无法防止看似可信但不正确的回答。

问题

A Romanian question‑answering demo that refuses any answer it cannot trace to a source document failed to detect invented answers that were built from true parts of the source.

结果

The model alone invented answers in 4 of 12 cases where evidence was removed; behind gates B, C and D it invented 6 of 12. All 24 invented answers passed the gates, showing that provenance checks alone cannot detect plausible but incorrect answers.

工作流

  • 1. Build 12 question pairs: one with the answer paragraph and one with the paragraph removed.
  • 2. Run the model alone on all 24 items.
  • 3. Run the model behind gates B, C and D on all 24 items.
  • 4. Record counts of correct answers, “not in text” responses, and invented answers.
  • 5. Analyze why invented answers passed the gates.
  • 6. Publish results and source code for reproducibility.

证据

The model invented an answer in 4 of 12 cases where the evidence had been removed.

来源

Gates B, C and D refused 0 of 24 items.

来源

The gates check provenance: whether the words came from the document. They do not check relevance: whether the sentence answers the question that was asked.

来源

The gated mode also uses the demo's own prompt, which is worded differently from the bare prompt.

来源

  • 记录日期: 2026-10-07T14:55:57.745Z