RetryBudget-24: When the Right Retry Decision Still Fails the JSON Contract

说明与用途

实验比较了 Gemini 2.5 Flash 和 Claude Haiku 4.5 在需要正确 JSON 输出的重试策略决策任务中的表现。模型经常返回错误的动作并将 JSON 包装在 Markdown 中，导致严格准确率低。

问题

An AI component must decide the correct retry action and produce a JSON contract that a client can parse, but models often output incorrect actions or malformed JSON.

结果

Gemini achieved 16/24 strict score (22/24 valid JSON) and 4/12 correct pair matches; Haiku achieved 5/24 strict score (9/24 valid JSON) and 1/12 correct pair matches. Post-hoc fence removal improved scores to 17/24 (Gemini) and 14/24 (Haiku).

工作流

• Define 12 pairs of synthetic cases with a single varying input field.
• Freeze prompts, oracle, and scorer before model calls.
• Run each case in a separate chat session with max_tokens=512, seed=0, temperature=0 (unsupported, dropped).
• Collect model outputs and compare against deterministic oracle for exact correctness.
• Separate strict JSON correctness from relaxed correctness by stripping outer Markdown fences.
• Analyze errors per policy boundary (budget, idempotency, backoff).
• Document unintended second experiment due to execution-control mistake and its impact on call counts and cost.
• Publish benchmark artifacts and results on Kaggle.
• Plan future experiments with strict vs. constrained-output conditions and platform build behavior verification.

证据

The experiment used two models available through Kaggle's Model Proxy: google/gemini-2.5-flash and anthropic/claude-haiku-4-5@20251001.

来源

The strict gap looks large. Inspecting the responses changes what can reasonably be concluded from it.

来源

Both models chose RETRY on the 299 ms budget example, although 100 + 200 exceeds 299.

来源