Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

We Tested Our Own AI Safety Checks. They Caught Nothing.

Описание и применение

Демонстрационная система вопросов‑ответов на румынском языке, использующая четыре проверки происхождения, не смогла обнаружить 6 из 12 выдуманных ответов, что показывает, что проверка только происхождения не защищает от правдоподобных, но неверных ответов.

Задача

A Romanian question‑answering demo that refuses any answer it cannot trace to a source document failed to detect invented answers that were built from true parts of the source.

Результат

The model alone invented answers in 4 of 12 cases where evidence was removed; behind gates B, C and D it invented 6 of 12. All 24 invented answers passed the gates, showing that provenance checks alone cannot detect plausible but incorrect answers.

Рабочий процесс

  • 1. Build 12 question pairs: one with the answer paragraph and one with the paragraph removed.
  • 2. Run the model alone on all 24 items.
  • 3. Run the model behind gates B, C and D on all 24 items.
  • 4. Record counts of correct answers, “not in text” responses, and invented answers.
  • 5. Analyze why invented answers passed the gates.
  • 6. Publish results and source code for reproducibility.

Доказательства

The model invented an answer in 4 of 12 cases where the evidence had been removed.

Источник

Gates B, C and D refused 0 of 24 items.

Источник

The gates check provenance: whether the words came from the document. They do not check relevance: whether the sentence answers the question that was asked.

Источник

The gated mode also uses the demo's own prompt, which is worded differently from the bare prompt.

Источник

  • Дата записи: 2026-10-07T14:55:57.745Z