I paid 9x more for the “smarter” LLM. It got exactly the same answers right.

Description and use

Choosing an LLM by token price can mislead; real cost is money paid for successful tasks, including retries and hidden reasoning.

Problem

Choosing an LLM based on token pricing can mislead; the real cost is the money paid for successful tasks, including retries and hidden reasoning.

Outcome

The flagship model’s higher token usage and reasoning output caused a 4.9x–9.2x cost increase per successful task compared to the cheaper model, even though both achieved 100% success within three attempts.

Workflow

• Clone the benchmark repo
• Install dependencies
• Run estimate to preview cost
• Set API keys for chosen providers
• Execute runner with desired models and budget
• Review generated report for cost per success and retry rate

Evidence

Both models eventually got every item right. The flagship cost 4.9x as much per success on everyday tasks and 9.2x on the hard ones.

Source

The benchmark had bugs too ... I fixed them before publishing

Source

Python 3.11+, bring your own keys. Nothing is sent anywhere except to the providers you choose.

Source