Your AI Agent Has a Burn Rate — So I Benchmarked the Cost of a Correct Answer

Description and use

The benchmark measures the cost per correct answer for agents, showing that tiered routing can achieve 95%+ accuracy at 29% lower cost than a large‑model‑only approach.

Problem

Production agents incur unpredictable costs because each tool call, retry, and retrieved document adds a priced round‑trip, yet teams rarely compute the cost per correct answer.

Outcome

Gemini 2.5 Flash achieved 100% accuracy on all five live tasks, while GPT‑OSS 120B failed a simple extraction. Tiered routing achieved 100% accuracy at 29% lower cost than a large‑model‑only strategy, demonstrating the value of routing policies with an accuracy floor.

Workflow

• Define 24 deterministic agent tasks (easy, medium, hard).
• Deploy first five easy tasks on Kaggle with five hosted models.
• Run tasks and record pass/fail results.
• Build local simulation of all 24 tasks with four routing strategies using tokenecon.
• Calculate accuracy, total cost, and cost per correct answer for each strategy.
• Analyze results to identify cost‑effective routing policies.
• Plan to publish medium and hard tiers on Kaggle next.

Evidence

A single model call has a predictable price. A looping agent does not. Every tool call, retry, and retrieved document adds another priced round‑trip, and most teams find out the total from the cloud bill at the end of the month.

Source

I built a benchmark that asks a different question. Not "what does the model cost?" — but "what does each correct answer cost?"

Source

The first five easy tasks are live on Kaggle now, run against five hosted models: Gemini 2.5 Flash, Gemini 3.7 Flash, DeepSeek R1, GPT-OSS 120B, and Qwen3 235B.

Source

Tiered routing matched the big model's accuracy at 29% lower cost — easy tasks went to the cheap tier, hard ones escalated.

Source

With a ≥95% accuracy requirement, tiered routing is the cheapest viable strategy — and that floor is the piece most cost dashboards are missing.

Source