RetryBudget-24: When the Right Retry Decision Still Fails the JSON Contract

Описание и применение

Эксперимент сравнивает Gemini 2.5 Flash и Claude Haiku 4.5 в задаче принятия решения о повторной попытке и генерации корректного JSON‑контракта. Модели часто возвращают неверные действия и оборачивают JSON в Markdown, что приводит к низкой точности строгих оценок.

Задача

An AI component must decide the correct retry action and produce a JSON contract that a client can parse, but models often output incorrect actions or malformed JSON.

Результат

Gemini achieved 16/24 strict score (22/24 valid JSON) and 4/12 correct pair matches; Haiku achieved 5/24 strict score (9/24 valid JSON) and 1/12 correct pair matches. Post-hoc fence removal improved scores to 17/24 (Gemini) and 14/24 (Haiku).

Рабочий процесс

• Define 12 pairs of synthetic cases with a single varying input field.
• Freeze prompts, oracle, and scorer before model calls.
• Run each case in a separate chat session with max_tokens=512, seed=0, temperature=0 (unsupported, dropped).
• Collect model outputs and compare against deterministic oracle for exact correctness.
• Separate strict JSON correctness from relaxed correctness by stripping outer Markdown fences.
• Analyze errors per policy boundary (budget, idempotency, backoff).
• Document unintended second experiment due to execution-control mistake and its impact on call counts and cost.
• Publish benchmark artifacts and results on Kaggle.
• Plan future experiments with strict vs. constrained-output conditions and platform build behavior verification.

Доказательства

The experiment used two models available through Kaggle's Model Proxy: google/gemini-2.5-flash and anthropic/claude-haiku-4-5@20251001.

Источник

The strict gap looks large. Inspecting the responses changes what can reasonably be concluded from it.

Источник

Both models chose RETRY on the 299 ms budget example, although 100 + 200 exceeds 299.

Источник