Tīrtha Bench: the model data coders actually need (and what we're building next)

Описание и применение

Тест Tīrtha Bench сравнивает 68 моделей по 800 задачам вызова инструментов, учитывая точность, стоимость за правильный ответ и влияние хоста. Результаты доступны на tirtha.ai/bench, где каждая модель имеет собственную страницу с подробными данными.

Задача

Developers lack a single source to compare language models on tool‑calling performance, cost, speed, and host variability.

Результат

The benchmark shows significant host‑dependent performance (e.g., Llama 3.3 70B: 93.4% vs 69.5%) and wide cost‑per‑correct‑answer ranges ($0.09–$5.63 per 1,000 correct calls).

Рабочий процесс

• 1. Collect 68 models. 2. Run each on 800 tool‑calling tasks (400 single, 200 multi‑tool, 200 parallel) plus 120 trap prompts. 3. Score all runs with a consistent checker. 4. Compute accuracy, cost per 1,000 correct calls, speed, and trap success. 5. Publish results on tirtha.ai/bench with per‑model pages and raw data.
• 6. Allow external users to reproduce runs and compare results.

Доказательства

We could not find that answered in one place, so we built it.

Источник

68 models tested, 8 on the same shared test, 800 tasks and 120 traps, same scorer

Источник

Same host matters: Llama 3.3 70B 93.4% on DeepInfra, 69.5% on SambaNova

Источник

$0.09 to $5.63 per 1,000 correct calls

Источник

Mikhail Maslenik ran our 800 requests on his own machine and scored 90.6% vs our 91.0%

Источник