To Retry or Not to Retry? That Is the Question

说明与用途

该基准测试评估 AI 模型在考虑上下文后决定是否重试 HTTP 请求的能力，揭示了系统性弱点，尤其是在“Unsafe Payment Retry”场景中。

问题

Determine whether an AI model correctly decides to retry HTTP requests based on context such as idempotency, status codes, and headers.

结果

GPT‑5.6 Sol achieved 13.67/14 average, Gemini 3.7 Flash 13.00, Gemini 3.1 Pro Preview 12.33, GLM‑5 12.33, Claude Sonnet 5 12.00, GPT‑5.6 Luna 12.00, DeepSeek‑R1 10.33, Claude Haiku 4.5 9.00. Models consistently failed on the Unsafe Payment Retry scenario (only 25% correct).

工作流

• Prepare 14 failure scenarios with request, response, and context details
• Ask models to output Decision (YES/NO/YES_AFTER_DELAY) and a one‑sentence Reason
• Collect decisions, compute accuracy per model
• Run benchmark three times to assess stability
• Analyze which scenarios cause systematic errors
• Report findings and model rankings

证据

The best model, with a score of 1.00, was GPT-5.6 Sol, followed by Gemini 3.7 Flash with one mistake.

来源

Most of the models made a mistake in the **Unsafe Payment Retry** task.

来源

The result was completely consistent across runs. GPT-5.6 Sol and Gemini 3.1 Pro Preview got it right all three times, while the other six models missed it in every run.

来源