To Retry or Not to Retry? That Is the Question

Description and use

A 14‑scenario benchmark was built to test AI models’ ability to decide whether to retry HTTP requests based on status, method, and context. GPT‑5.6 Sol performed best; most models failed consistently on the Unsafe Payment Retry scenario due to misreading Retry‑After.

Problem

Determine whether an HTTP request should be retried based on status code, method, idempotency, and context.

Outcome

GPT-5.6 Sol achieved 13.67/14 average, Gemini 3.7 Flash 13.00, others ranged 9–12. Most models failed consistently on Unsafe Payment Retry due to misinterpreting Retry-After.

Workflow

• Prepare 14 API failure scenarios with context
• Define expected decision (YES/NO/YES_AFTER_DELAY) for each
• Run each model on all scenarios
• Collect Decision and Reason outputs
• Score based on Decision correctness
• Repeat benchmark three times for stability
• Analyze results and identify systematic errors

Evidence

I prepared several API failure scenarios containing information about the request, the response, and some additional context.

Source

The best model, with a score of 1.00, was GPT-5.6 Sol, followed by Gemini 3.7 Flash with one mistake.

Source

Most of the models made a mistake in the **Unsafe Payment Retry** task.

Source