To Retry or Not to Retry? That Is the Question

Описание и применение

Автор создал бенчмарк, оценивающий, как ИИ принимает решения о повторных запросах к API, учитывая контекст, такой как idempotency и заголовки. Лучшие модели — GPT‑5.6 Sol и Gemini 3.7 Flash, но большинство моделей ошибаются в сложных сценариях, особенно в «Unsafe Payment Retry».

Задача

How to determine when an AI model should retry an HTTP request based on context such as idempotency, status codes, and headers.

Результат

GPT‑5.6 Sol achieved the highest score (13.67/14) with consistent correct decisions, especially in the Unsafe Payment Retry scenario. Gemini 3.7 Flash and Gemini 3.1 Pro Preview followed closely. Most models struggled with scenarios requiring contextual understanding beyond HTTP signals, revealing a systematic trap.

Рабочий процесс

• 1. Define 14 API failure scenarios with context (method, status, headers, idempotency).
• 2. Specify expected decision (YES, NO, YES_AFTER_DELAY) for each scenario.
• 3. Run selected AI models (GPT‑5.6 Sol, GPT‑5.6 Luna, Gemini 3.7 Flash, Gemini 3.1 Pro Preview, Claude Sonnet 5, Claude Haiku 4.5, GLM‑5, DeepSeek‑R1) on all scenarios.
• 4. Collect model outputs in the format: Decision: … Reason: …
• 5. Score models based only on Decision correctness.
• 6. Repeat the benchmark three times to assess stability.
• 7. Analyze results, noting systematic errors and stability patterns.

Доказательства

The idea is simple. I prepared several API failure scenarios containing information about the request, the response, and some additional context.

Источник

We can see that the best model, with a score of 1.00, was GPT-5.6 Sol, followed by Gemini 3.7 Flash with one mistake.

Источник

Most of the models made a mistake in the **Unsafe Payment Retry** task.

Источник

I ran the same benchmark two more times to check model stability and collect more data.

Источник