To Retry or Not to Retry? That Is the Question

Описание и применение

Бенчмарк проверяет, как ИИ принимает решения о повторных запросах к API, учитывая контекст, и выявляет системные слабости моделей.

Задача

Determine whether an AI model correctly decides to retry HTTP requests based on context such as idempotency, status codes, and headers.

Результат

GPT‑5.6 Sol achieved 13.67/14 average, Gemini 3.7 Flash 13.00, Gemini 3.1 Pro Preview 12.33, GLM‑5 12.33, Claude Sonnet 5 12.00, GPT‑5.6 Luna 12.00, DeepSeek‑R1 10.33, Claude Haiku 4.5 9.00. Models consistently failed on the Unsafe Payment Retry scenario (only 25% correct).

Рабочий процесс

• Prepare 14 failure scenarios with request, response, and context details
• Ask models to output Decision (YES/NO/YES_AFTER_DELAY) and a one‑sentence Reason
• Collect decisions, compute accuracy per model
• Run benchmark three times to assess stability
• Analyze which scenarios cause systematic errors
• Report findings and model rankings

Доказательства

The best model, with a score of 1.00, was GPT-5.6 Sol, followed by Gemini 3.7 Flash with one mistake.

Источник

Most of the models made a mistake in the **Unsafe Payment Retry** task.

Источник

The result was completely consistent across runs. GPT-5.6 Sol and Gemini 3.1 Pro Preview got it right all three times, while the other six models missed it in every run.

Источник