The Agent Finished the Job. The Benchmark Gave It Zero.

Description and use

The agent completed the task, yet the benchmark awarded zero due to final JSON format errors, despite correct business progress and no duplicate writes.

Problem

An agent must decide how to handle a support ticket when the acknowledgement is lost, leading to uncertainty about whether the ticket was created.

Outcome

Both tested models completed all business‑progress cases and avoided duplicate writes, but one model scored zero under the strict evaluation contract due to final JSON format violations.

Workflow

• Submit support ticket
• System creates ticket
• Connection breaks before acknowledgement
• Agent times out
• Agent retrieves original result or recovers work
• Agent returns plain‑JSON final response

Evidence

The benchmark has 24 synthetic cases, four in each of six families

Source

Both tested models completed every case that required business progress

Source

One model nevertheless scored <strong>zero</strong> under the frozen evaluation contract.

Source

The 34 progress episodes per model are the 17 cases requiring progress, each run twice.

Source