To Retry or Not to Retry? That Is the Question

Описание и применение

Бенчмарк проверяет, правильно ли модели ИИ решают, стоит ли повторять HTTP‑запросы, учитывая контекст, такой как идемпотентность и заголовки. На 14 сценариях GPT‑5.6 Sol показал лучший результат, а большинство моделей ошибались в случае «Unsafe Payment Retry».

Задача

Determine whether an AI model correctly decides to retry HTTP requests based on context such as idempotency, status codes, and headers.

Результат

GPT‑5.6 Sol achieved 13.67/14 average, Gemini 3.7 Flash 13.00, Gemini 3.1 Pro Preview 12.33, GLM‑5 12.33, Claude Sonnet 5 12.00, GPT‑5.6 Luna 12.00, DeepSeek‑R1 10.33, Claude Haiku 4.5 9.00. The Unsafe Payment Retry scenario was consistently misinterpreted by most models.

Рабочий процесс

• Prepare 14 failure scenarios with request, response, and context
• Ask each model to output Decision and Reason
• Collect decisions, compute accuracy per model
• Run benchmark three times to assess stability
• Analyze which scenarios cause systematic errors

Доказательства

The best model, with a score of 1.00, was GPT-5.6 Sol, followed by Gemini 3.7 Flash with one mistake.

Источник

Most of the models made a mistake in the **Unsafe Payment Retry** task.

Источник

The benchmark used 14 scenarios, each based on an actual task used in the Kaggle benchmark.

Источник