We Tested Our Own AI Safety Checks. They Caught Nothing.

Description and use

A Romanian QA demo with provenance gates failed to detect 6 of 12 invented answers, revealing that provenance‑only checks miss plausible but incorrect responses.

Problem

A Romanian question‑answering demo that refuses any answer it cannot trace to a source document failed to detect invented answers that were built from true parts of the source.

Outcome

The model alone invented answers in 4 of 12 cases where evidence was removed; behind gates B, C and D it invented 6 of 12. All 24 invented answers passed the gates, showing that provenance checks alone cannot detect plausible but incorrect answers.

Workflow

• 1. Build 12 question pairs: one with the answer paragraph and one with the paragraph removed.
• 2. Run the model alone on all 24 items.
• 3. Run the model behind gates B, C and D on all 24 items.
• 4. Record counts of correct answers, “not in text” responses, and invented answers.
• 5. Analyze why invented answers passed the gates.
• 6. Publish results and source code for reproducibility.

Evidence

The model invented an answer in 4 of 12 cases where the evidence had been removed.

Source

Gates B, C and D refused 0 of 24 items.

Source

The gates check provenance: whether the words came from the document. They do not check relevance: whether the sentence answers the question that was asked.

Source

The gated mode also uses the demo's own prompt, which is worded differently from the bare prompt.

Source