I paid 9x more for the “smarter” LLM. It got exactly the same answers right.

Описание и применение

Выбор модели по цене токенов может вводить в заблуждение: реальная стоимость – это деньги, потраченные за успешные задачи, включая повторные попытки и скрытое рассуждение.

Задача

Choosing an LLM based on token pricing can mislead; the real cost is the money paid for successful tasks, including retries and hidden reasoning.

Результат

The flagship model’s higher token usage and reasoning output caused a 4.9x–9.2x cost increase per successful task compared to the cheaper model, even though both achieved 100% success within three attempts.

Рабочий процесс

• Clone the benchmark repo
• Install dependencies
• Run estimate to preview cost
• Set API keys for chosen providers
• Execute runner with desired models and budget
• Review generated report for cost per success and retry rate

Доказательства

Both models eventually got every item right. The flagship cost 4.9x as much per success on everyday tasks and 9.2x on the hard ones.

Источник

The benchmark had bugs too ... I fixed them before publishing

Источник

Python 3.11+, bring your own keys. Nothing is sent anywhere except to the providers you choose.

Источник