To Retry or Not to Retry? That Is the Question.

说明与用途

作者创建了14个API失败场景的基准，用于评估AI模型的重试决策。GPT‑5.6 Sol表现最佳，其他模型大多在“Unsafe Payment Retry”场景中误解了Retry‑After头。

问题

Determine whether an AI model can correctly decide to retry, not retry, or retry after a delay for various HTTP failure scenarios, ensuring that retry logic aligns with business rules such as idempotency and safety.

结果

GPT‑5.6 Sol achieved the highest accuracy (average 13.67/14), followed by Gemini 3.7 Flash (13.00/14). Most models struggled with the "Unsafe Payment Retry" scenario, incorrectly following the Retry‑After header despite lack of idempotency.

工作流

• 1. Define 14 failure scenarios with request, response, and context details.
• 2. Specify expected decision (YES/NO/YES_AFTER_DELAY) and explanation for each scenario.
• 3. Format model prompts to return "Decision: … Reason: …".
• 4. Run selected AI models (GPT‑5.6 Sol, GPT‑5.6 Luna, Gemini 3.7 Flash, Gemini 3.1 Pro Preview, Claude Sonnet 5, Claude Haiku 4.5, GLM‑5, DeepSeek‑R1) across three benchmark runs.
• 5. Record decisions and compute accuracy per model.
• 6. Analyze common error patterns (e.g., misinterpreting Retry‑After with non‑idempotent payments).
• 7. Publish results and insights for developers to inform retry policy design.

证据

I prepared several API failure scenarios containing information about the request, the response, and some additional context.

来源

The model has to return two things: a decision: YES, NO, or YES_AFTER_DELAY; a one-sentence explanation of why

来源

We can see that the best model, with a score of 1.00, was GPT-5.6 Sol, followed by Gemini 3.7 Flash with one mistake.

来源

Most of the models made a mistake in the Unsafe Payment Retry task.

来源