To Retry or Not to Retry? That Is the Question.

Описание и применение

Автор создал набор из 14 сценариев отказов API, чтобы проверить, как ИИ‑модели принимают решения о повторных запросах. GPT‑5.6 Sol показал наилучшие результаты, а большинство моделей ошибались в сценарии «Unsafe Payment Retry» из‑за неверного применения заголовка Retry‑After.

Задача

Determine whether an AI model can correctly decide to retry, not retry, or retry after a delay for various HTTP failure scenarios, ensuring that retry logic aligns with business rules such as idempotency and safety.

Результат

GPT‑5.6 Sol achieved the highest accuracy (average 13.67/14), followed by Gemini 3.7 Flash (13.00/14). Most models struggled with the "Unsafe Payment Retry" scenario, incorrectly following the Retry‑After header despite lack of idempotency.

Рабочий процесс

• 1. Define 14 failure scenarios with request, response, and context details.
• 2. Specify expected decision (YES/NO/YES_AFTER_DELAY) and explanation for each scenario.
• 3. Format model prompts to return "Decision: … Reason: …".
• 4. Run selected AI models (GPT‑5.6 Sol, GPT‑5.6 Luna, Gemini 3.7 Flash, Gemini 3.1 Pro Preview, Claude Sonnet 5, Claude Haiku 4.5, GLM‑5, DeepSeek‑R1) across three benchmark runs.
• 5. Record decisions and compute accuracy per model.
• 6. Analyze common error patterns (e.g., misinterpreting Retry‑After with non‑idempotent payments).
• 7. Publish results and insights for developers to inform retry policy design.

Доказательства

I prepared several API failure scenarios containing information about the request, the response, and some additional context.

Источник

The model has to return two things: a decision: YES, NO, or YES_AFTER_DELAY; a one-sentence explanation of why

Источник

We can see that the best model, with a score of 1.00, was GPT-5.6 Sol, followed by Gemini 3.7 Flash with one mistake.

Источник

Most of the models made a mistake in the Unsafe Payment Retry task.

Источник