To Retry or Not to Retry? That Is the Question

说明与用途

创建了14个场景来测试AI模型在考虑状态码、方法和上下文时是否应该重试HTTP请求。GPT‑5.6 Sol表现最佳，绝大多数模型在“Unsafe Payment Retry”场景中持续错误，原因是误解了Retry‑After头。

问题

Determine whether an HTTP request should be retried based on status code, method, idempotency, and context.

结果

GPT-5.6 Sol achieved 13.67/14 average, Gemini 3.7 Flash 13.00, others ranged 9–12. Most models failed consistently on Unsafe Payment Retry due to misinterpreting Retry-After.

工作流

• Prepare 14 API failure scenarios with context
• Define expected decision (YES/NO/YES_AFTER_DELAY) for each
• Run each model on all scenarios
• Collect Decision and Reason outputs
• Score based on Decision correctness
• Repeat benchmark three times for stability
• Analyze results and identify systematic errors

证据

I prepared several API failure scenarios containing information about the request, the response, and some additional context.

来源

The best model, with a score of 1.00, was GPT-5.6 Sol, followed by Gemini 3.7 Flash with one mistake.

来源

Most of the models made a mistake in the **Unsafe Payment Retry** task.

来源