Your AI Agent Has a Burn Rate — So I Benchmarked the Cost of a Correct Answer

说明与用途

该基准衡量代理的正确答案成本，显示分层路由可在保持95%+准确率的同时，比仅使用大模型降低29%的成本。

问题

Production agents incur unpredictable costs because each tool call, retry, and retrieved document adds a priced round‑trip, yet teams rarely compute the cost per correct answer.

结果

Gemini 2.5 Flash achieved 100% accuracy on all five live tasks, while GPT‑OSS 120B failed a simple extraction. Tiered routing achieved 100% accuracy at 29% lower cost than a large‑model‑only strategy, demonstrating the value of routing policies with an accuracy floor.

工作流

• Define 24 deterministic agent tasks (easy, medium, hard).
• Deploy first five easy tasks on Kaggle with five hosted models.
• Run tasks and record pass/fail results.
• Build local simulation of all 24 tasks with four routing strategies using tokenecon.
• Calculate accuracy, total cost, and cost per correct answer for each strategy.
• Analyze results to identify cost‑effective routing policies.
• Plan to publish medium and hard tiers on Kaggle next.

证据

A single model call has a predictable price. A looping agent does not. Every tool call, retry, and retrieved document adds another priced round‑trip, and most teams find out the total from the cloud bill at the end of the month.

来源

I built a benchmark that asks a different question. Not "what does the model cost?" — but "what does each correct answer cost?"

来源

The first five easy tasks are live on Kaggle now, run against five hosted models: Gemini 2.5 Flash, Gemini 3.7 Flash, DeepSeek R1, GPT-OSS 120B, and Qwen3 235B.

来源

Tiered routing matched the big model's accuracy at 29% lower cost — easy tasks went to the cheap tier, hard ones escalated.

来源

With a ≥95% accuracy requirement, tiered routing is the cheapest viable strategy — and that floor is the piece most cost dashboards are missing.

来源