RetryBudget-24: When the Right Retry Decision Still Fails the JSON Contract

Description and use

The experiment compares Gemini 2.5 Flash and Claude Haiku 4.5 on a retry‑policy decision task requiring correct JSON output. Models frequently produce incorrect actions and wrap JSON in Markdown, resulting in low strict accuracy.

Problem

An AI component must decide the correct retry action and produce a JSON contract that a client can parse, but models often output incorrect actions or malformed JSON.

Outcome

Gemini achieved 16/24 strict score (22/24 valid JSON) and 4/12 correct pair matches; Haiku achieved 5/24 strict score (9/24 valid JSON) and 1/12 correct pair matches. Post-hoc fence removal improved scores to 17/24 (Gemini) and 14/24 (Haiku).

Workflow

• Define 12 pairs of synthetic cases with a single varying input field.
• Freeze prompts, oracle, and scorer before model calls.
• Run each case in a separate chat session with max_tokens=512, seed=0, temperature=0 (unsupported, dropped).
• Collect model outputs and compare against deterministic oracle for exact correctness.
• Separate strict JSON correctness from relaxed correctness by stripping outer Markdown fences.
• Analyze errors per policy boundary (budget, idempotency, backoff).
• Document unintended second experiment due to execution-control mistake and its impact on call counts and cost.
• Publish benchmark artifacts and results on Kaggle.
• Plan future experiments with strict vs. constrained-output conditions and platform build behavior verification.

Evidence

The experiment used two models available through Kaggle's Model Proxy: google/gemini-2.5-flash and anthropic/claude-haiku-4-5@20251001.

Source

The strict gap looks large. Inspecting the responses changes what can reasonably be concluded from it.

Source

Both models chose RETRY on the 299 ms budget example, although 100 + 200 exceeds 299.

Source