To Retry or Not to Retry? That Is the Question

Description and use

The benchmark evaluates AI models’ decisions to retry HTTP requests based on context, revealing systematic weaknesses, especially in the Unsafe Payment Retry scenario.

Problem

Determine whether an AI model correctly decides to retry HTTP requests based on context such as idempotency, status codes, and headers.

Outcome

GPT‑5.6 Sol achieved 13.67/14 average, Gemini 3.7 Flash 13.00, Gemini 3.1 Pro Preview 12.33, GLM‑5 12.33, Claude Sonnet 5 12.00, GPT‑5.6 Luna 12.00, DeepSeek‑R1 10.33, Claude Haiku 4.5 9.00. Models consistently failed on the Unsafe Payment Retry scenario (only 25% correct).

Workflow

• Prepare 14 failure scenarios with request, response, and context details
• Ask models to output Decision (YES/NO/YES_AFTER_DELAY) and a one‑sentence Reason
• Collect decisions, compute accuracy per model
• Run benchmark three times to assess stability
• Analyze which scenarios cause systematic errors
• Report findings and model rankings

Evidence

The best model, with a score of 1.00, was GPT-5.6 Sol, followed by Gemini 3.7 Flash with one mistake.

Source

Most of the models made a mistake in the **Unsafe Payment Retry** task.

Source

The result was completely consistent across runs. GPT-5.6 Sol and Gemini 3.1 Pro Preview got it right all three times, while the other six models missed it in every run.

Source