Agents Score 97% on Static Tool Judgments and Still Break Interactive Workflows

Description and use

<p>If you ask an LLM in a static prompt whether it should refund a customer's duplicate charge before checking the payment status, almost every frontier model passes the test. It outputs a clean refusal or defers the call until the record is verified.</p> <p>Then you put the exact same model inside an interactive tool loop and watch it fire the refund call anyway.</p> <p>Two papers from the October 7, 2026 arXiv batch isolate how this failure happens. Both teams separate what an agent knows from what it executes, and both land on the same engineering fix. Prompt rules and static policy evals b

Problem

<p>If you ask an LLM in a static prompt whether it should refund a customer's duplicate charge before checking the payment status, almost every frontier model passes the test. It outputs a clean refusal or defers the call until the record is verified.</p> <p>Then you put the exact same model inside an interactive tool loop and watch it fire the refund call anyway.</p> <p>Two papers from the October 7, 2026 arXiv batch isolate how this failure happens. Both teams separate what an agent knows from what it executes, and both land on the same engineering fix. Prompt rules and static policy evals b

Outcome

<p>If you ask an LLM in a static prompt whether it should refund a customer's duplicate charge before checking the payment status, almost every frontier model passes the test. It outputs a clean refusal or defers the call until the record is verified.</p> <p>Then you put the exact same model inside an interactive tool loop and watch it fire the refund call anyway.</p> <p>Two papers from the October 7, 2026 arXiv batch isolate how this failure happens. Both teams separate what an agent knows from what it executes, and both land on the same engineering fix. Prompt rules and static policy evals b

Evidence

Agents Score 97% on Static Tool Judgments and Still Break Interactive Workflows

Source