The Agent Finished the Job. The Benchmark Gave It Zero.

Описание и применение

Агент завершил работу, но оценка дала ноль из‑за ошибок формата JSON, хотя бизнес‑операции выполнены без дублирования.

Задача

An agent must decide how to handle a support ticket when the acknowledgement is lost, leading to uncertainty about whether the ticket was created.

Результат

Both tested models completed all business‑progress cases and avoided duplicate writes, but one model scored zero under the strict evaluation contract due to final JSON format violations.

Рабочий процесс

• Submit support ticket
• System creates ticket
• Connection breaks before acknowledgement
• Agent times out
• Agent retrieves original result or recovers work
• Agent returns plain‑JSON final response

Доказательства

The benchmark has 24 synthetic cases, four in each of six families

Источник

Both tested models completed every case that required business progress

Источник

One model nevertheless scored <strong>zero</strong> under the frozen evaluation contract.

Источник

The 34 progress episodes per model are the 17 cases requiring progress, each run twice.

Источник