To Retry or Not to Retry? That Is the Question.

Description and use

The author built a 14‑scenario benchmark to evaluate AI models’ retry decisions for API failures. GPT‑5.6 Sol performed best, while most models misinterpreted the Unsafe Payment Retry scenario due to the Retry‑After header.

Problem

Determine whether an AI model can correctly decide to retry, not retry, or retry after a delay for various HTTP failure scenarios, ensuring that retry logic aligns with business rules such as idempotency and safety.

Outcome

GPT‑5.6 Sol achieved the highest accuracy (average 13.67/14), followed by Gemini 3.7 Flash (13.00/14). Most models struggled with the "Unsafe Payment Retry" scenario, incorrectly following the Retry‑After header despite lack of idempotency.

Workflow

• 1. Define 14 failure scenarios with request, response, and context details.
• 2. Specify expected decision (YES/NO/YES_AFTER_DELAY) and explanation for each scenario.
• 3. Format model prompts to return "Decision: … Reason: …".
• 4. Run selected AI models (GPT‑5.6 Sol, GPT‑5.6 Luna, Gemini 3.7 Flash, Gemini 3.1 Pro Preview, Claude Sonnet 5, Claude Haiku 4.5, GLM‑5, DeepSeek‑R1) across three benchmark runs.
• 5. Record decisions and compute accuracy per model.
• 6. Analyze common error patterns (e.g., misinterpreting Retry‑After with non‑idempotent payments).
• 7. Publish results and insights for developers to inform retry policy design.

Evidence

I prepared several API failure scenarios containing information about the request, the response, and some additional context.

Source

The model has to return two things: a decision: YES, NO, or YES_AFTER_DELAY; a one-sentence explanation of why

Source

We can see that the best model, with a score of 1.00, was GPT-5.6 Sol, followed by Gemini 3.7 Flash with one mistake.

Source

Most of the models made a mistake in the Unsafe Payment Retry task.

Source