The check failed. Why did the AI remember the answer?

Description and use

When a check fails, the old logic replayed the unchecked answer into the next turn, causing the AI to remember it. The new logic stops this replay, keeping the answer out of subsequent context.

Problem

When a check on an AI assistant’s answer fails, the answer may still be stored and sent to the next model call, causing the assistant to “remember” an unverified response.

Outcome

The patch ensures that answers failing validation are not automatically included in future model calls, preventing the assistant from using unverified information as context.

Workflow

• 1. Configure a check that can pass, reject, or throw.
• 2. Run the assistant with a test model that returns a distinctive marker text.
• 3. Observe the conversation history sent to the model after the check outcome.
• 4. Compare old logic (replays answer on error) vs. fixed logic (does not replay).

Evidence

In our controlled comparison, the old logic raised an error and saved the answer anyway. The next model call received it.

Source

The fixed logic preserves that distinction. It does not declare the answer disproved, and it does not automatically admit the answer as context for later work.

Source