To Retry or Not to Retry? That Is the Question

Description and use

The author built a benchmark to evaluate AI models on retry decisions for API failures, revealing systematic weaknesses, especially in the Unsafe Payment Retry scenario, and highlighting differences in model stability across runs.

Problem

How to determine when an AI model should retry an HTTP request based on context such as idempotency, status codes, and headers.

Outcome

GPT‑5.6 Sol achieved the highest score (13.67/14) with consistent correct decisions, especially in the Unsafe Payment Retry scenario. Gemini 3.7 Flash and Gemini 3.1 Pro Preview followed closely. Most models struggled with scenarios requiring contextual understanding beyond HTTP signals, revealing a systematic trap.

Workflow

• 1. Define 14 API failure scenarios with context (method, status, headers, idempotency).
• 2. Specify expected decision (YES, NO, YES_AFTER_DELAY) for each scenario.
• 3. Run selected AI models (GPT‑5.6 Sol, GPT‑5.6 Luna, Gemini 3.7 Flash, Gemini 3.1 Pro Preview, Claude Sonnet 5, Claude Haiku 4.5, GLM‑5, DeepSeek‑R1) on all scenarios.
• 4. Collect model outputs in the format: Decision: … Reason: …
• 5. Score models based only on Decision correctness.
• 6. Repeat the benchmark three times to assess stability.
• 7. Analyze results, noting systematic errors and stability patterns.

Evidence

The idea is simple. I prepared several API failure scenarios containing information about the request, the response, and some additional context.

Source

We can see that the best model, with a score of 1.00, was GPT-5.6 Sol, followed by Gemini 3.7 Flash with one mistake.

Source

Most of the models made a mistake in the **Unsafe Payment Retry** task.

Source

I ran the same benchmark two more times to check model stability and collect more data.

Source