I paid 9x more for the “smarter” LLM. It got exactly the same answers right.

说明与用途

按令牌价格选择 LLM 可能误导：真正的成本是为成功任务支付的费用，包括重试和隐藏推理。

问题

Choosing an LLM based on token pricing can mislead; the real cost is the money paid for successful tasks, including retries and hidden reasoning.

结果

The flagship model’s higher token usage and reasoning output caused a 4.9x–9.2x cost increase per successful task compared to the cheaper model, even though both achieved 100% success within three attempts.

工作流

• Clone the benchmark repo
• Install dependencies
• Run estimate to preview cost
• Set API keys for chosen providers
• Execute runner with desired models and budget
• Review generated report for cost per success and retry rate

证据

Both models eventually got every item right. The flagship cost 4.9x as much per success on everyday tasks and 9.2x on the hard ones.

来源

The benchmark had bugs too ... I fixed them before publishing

来源

Python 3.11+, bring your own keys. Nothing is sent anywhere except to the providers you choose.

来源