To Retry or Not to Retry? That Is the Question

说明与用途

该基准测试 AI 模型是否能根据上下文（如幂等性和头信息）正确决定是否重试 HTTP 请求。GPT‑5.6 Sol 取得最佳成绩，绝大多数模型在“Unsafe Payment Retry”场景中失误。

问题

Determine whether an AI model correctly decides to retry HTTP requests based on context such as idempotency, status codes, and headers.

结果

GPT‑5.6 Sol achieved 13.67/14 average, Gemini 3.7 Flash 13.00, Gemini 3.1 Pro Preview 12.33, GLM‑5 12.33, Claude Sonnet 5 12.00, GPT‑5.6 Luna 12.00, DeepSeek‑R1 10.33, Claude Haiku 4.5 9.00. The Unsafe Payment Retry scenario was consistently misinterpreted by most models.

工作流

• Prepare 14 failure scenarios with request, response, and context
• Ask each model to output Decision and Reason
• Collect decisions, compute accuracy per model
• Run benchmark three times to assess stability
• Analyze which scenarios cause systematic errors

证据

The best model, with a score of 1.00, was GPT-5.6 Sol, followed by Gemini 3.7 Flash with one mistake.

来源

Most of the models made a mistake in the **Unsafe Payment Retry** task.

来源

The benchmark used 14 scenarios, each based on an actual task used in the Kaggle benchmark.

来源