To Retry or Not to Retry? That Is the Question

说明与用途

作者创建了一个基准测试，用于评估 AI 模型在 API 失败情况下的重试决策，发现系统性弱点，尤其是在“Unsafe Payment Retry”场景中，并突出了模型在多次运行中的稳定性差异。

问题

How to determine when an AI model should retry an HTTP request based on context such as idempotency, status codes, and headers.

结果

GPT‑5.6 Sol achieved the highest score (13.67/14) with consistent correct decisions, especially in the Unsafe Payment Retry scenario. Gemini 3.7 Flash and Gemini 3.1 Pro Preview followed closely. Most models struggled with scenarios requiring contextual understanding beyond HTTP signals, revealing a systematic trap.

工作流

• 1. Define 14 API failure scenarios with context (method, status, headers, idempotency).
• 2. Specify expected decision (YES, NO, YES_AFTER_DELAY) for each scenario.
• 3. Run selected AI models (GPT‑5.6 Sol, GPT‑5.6 Luna, Gemini 3.7 Flash, Gemini 3.1 Pro Preview, Claude Sonnet 5, Claude Haiku 4.5, GLM‑5, DeepSeek‑R1) on all scenarios.
• 4. Collect model outputs in the format: Decision: … Reason: …
• 5. Score models based only on Decision correctness.
• 6. Repeat the benchmark three times to assess stability.
• 7. Analyze results, noting systematic errors and stability patterns.

证据

The idea is simple. I prepared several API failure scenarios containing information about the request, the response, and some additional context.

来源

We can see that the best model, with a score of 1.00, was GPT-5.6 Sol, followed by Gemini 3.7 Flash with one mistake.

来源

Most of the models made a mistake in the **Unsafe Payment Retry** task.

来源

I ran the same benchmark two more times to check model stability and collect more data.

来源