To Retry or Not to Retry? That Is the Question

Описание и применение

Создано 14‑сценариев для проверки, будет ли модель корректно решать, стоит ли повторять запрос, учитывая статус, метод и контекст. GPT‑5.6 Sol показал лучший результат, а большинство моделей ошибались в задаче «Unsafe Payment Retry» из‑за неверного толкования заголовка Retry‑After.

Задача

Determine whether an HTTP request should be retried based on status code, method, idempotency, and context.

Результат

GPT-5.6 Sol achieved 13.67/14 average, Gemini 3.7 Flash 13.00, others ranged 9–12. Most models failed consistently on Unsafe Payment Retry due to misinterpreting Retry-After.

Рабочий процесс

• Prepare 14 API failure scenarios with context
• Define expected decision (YES/NO/YES_AFTER_DELAY) for each
• Run each model on all scenarios
• Collect Decision and Reason outputs
• Score based on Decision correctness
• Repeat benchmark three times for stability
• Analyze results and identify systematic errors

Доказательства

I prepared several API failure scenarios containing information about the request, the response, and some additional context.

Источник

The best model, with a score of 1.00, was GPT-5.6 Sol, followed by Gemini 3.7 Flash with one mistake.

Источник

Most of the models made a mistake in the **Unsafe Payment Retry** task.

Источник