To Retry or Not to Retry? That Is the Question

Description and use

The benchmark tests whether AI models correctly decide to retry HTTP requests based on context such as idempotency and headers. GPT‑5.6 Sol performed best, while most models failed on the Unsafe Payment Retry scenario.

Problem

Determine whether an AI model correctly decides to retry HTTP requests based on context such as idempotency, status codes, and headers.

Outcome

GPT‑5.6 Sol achieved 13.67/14 average, Gemini 3.7 Flash 13.00, Gemini 3.1 Pro Preview 12.33, GLM‑5 12.33, Claude Sonnet 5 12.00, GPT‑5.6 Luna 12.00, DeepSeek‑R1 10.33, Claude Haiku 4.5 9.00. The Unsafe Payment Retry scenario was consistently misinterpreted by most models.

Workflow

• Prepare 14 failure scenarios with request, response, and context
• Ask each model to output Decision and Reason
• Collect decisions, compute accuracy per model
• Run benchmark three times to assess stability
• Analyze which scenarios cause systematic errors

Evidence

The best model, with a score of 1.00, was GPT-5.6 Sol, followed by Gemini 3.7 Flash with one mistake.

Source

Most of the models made a mistake in the **Unsafe Payment Retry** task.

Source

The benchmark used 14 scenarios, each based on an actual task used in the Kaggle benchmark.

Source