Your AI Agent Has a Burn Rate — So I Benchmarked the Cost of a Correct Answer

Описание и применение

Бенчмарк измеряет стоимость правильного ответа агенту, показывая, что маршрутизация по уровням может достичь 95%+ точности при 29% меньших затратах, чем использование только большого модели.

Задача

Production agents incur unpredictable costs because each tool call, retry, and retrieved document adds a priced round‑trip, yet teams rarely compute the cost per correct answer.

Результат

Gemini 2.5 Flash achieved 100% accuracy on all five live tasks, while GPT‑OSS 120B failed a simple extraction. Tiered routing achieved 100% accuracy at 29% lower cost than a large‑model‑only strategy, demonstrating the value of routing policies with an accuracy floor.

Рабочий процесс

• Define 24 deterministic agent tasks (easy, medium, hard).
• Deploy first five easy tasks on Kaggle with five hosted models.
• Run tasks and record pass/fail results.
• Build local simulation of all 24 tasks with four routing strategies using tokenecon.
• Calculate accuracy, total cost, and cost per correct answer for each strategy.
• Analyze results to identify cost‑effective routing policies.
• Plan to publish medium and hard tiers on Kaggle next.

Доказательства

A single model call has a predictable price. A looping agent does not. Every tool call, retry, and retrieved document adds another priced round‑trip, and most teams find out the total from the cloud bill at the end of the month.

Источник

I built a benchmark that asks a different question. Not "what does the model cost?" — but "what does each correct answer cost?"

Источник

The first five easy tasks are live on Kaggle now, run against five hosted models: Gemini 2.5 Flash, Gemini 3.7 Flash, DeepSeek R1, GPT-OSS 120B, and Qwen3 235B.

Источник

Tiered routing matched the big model's accuracy at 29% lower cost — easy tasks went to the cheap tier, hard ones escalated.

Источник

With a ≥95% accuracy requirement, tiered routing is the cheapest viable strategy — and that floor is the piece most cost dashboards are missing.

Источник