The Agent Finished the Job. The Benchmark Gave It Zero.

说明与用途

代理完成了工作，但由于最终JSON格式错误，基准评估为零，尽管业务进展正确且没有重复写入。

问题

An agent must decide how to handle a support ticket when the acknowledgement is lost, leading to uncertainty about whether the ticket was created.

结果

Both tested models completed all business‑progress cases and avoided duplicate writes, but one model scored zero under the strict evaluation contract due to final JSON format violations.

工作流

• Submit support ticket
• System creates ticket
• Connection breaks before acknowledgement
• Agent times out
• Agent retrieves original result or recovers work
• Agent returns plain‑JSON final response

证据

The benchmark has 24 synthetic cases, four in each of six families

来源

Both tested models completed every case that required business progress

来源

One model nevertheless scored <strong>zero</strong> under the frozen evaluation contract.

来源

The 34 progress episodes per model are the 17 cases requiring progress, each run twice.

来源