Tīrtha Bench: the model data coders actually need (and what we're building next)

说明与用途

Tīrtha Bench 对 68 个模型进行 800 个工具调用任务的评估，报告准确率、每个正确答案的成本、速度和主机影响。结果公开在 tirtha.ai/bench，每个模型都有自己的页面和原始数据。

问题

Developers lack a single source to compare language models on tool‑calling performance, cost, speed, and host variability.

结果

The benchmark shows significant host‑dependent performance (e.g., Llama 3.3 70B: 93.4% vs 69.5%) and wide cost‑per‑correct‑answer ranges ($0.09–$5.63 per 1,000 correct calls).

工作流

• 1. Collect 68 models. 2. Run each on 800 tool‑calling tasks (400 single, 200 multi‑tool, 200 parallel) plus 120 trap prompts. 3. Score all runs with a consistent checker. 4. Compute accuracy, cost per 1,000 correct calls, speed, and trap success. 5. Publish results on tirtha.ai/bench with per‑model pages and raw data.
• 6. Allow external users to reproduce runs and compare results.

证据

We could not find that answered in one place, so we built it.

来源

68 models tested, 8 on the same shared test, 800 tasks and 120 traps, same scorer

来源

Same host matters: Llama 3.3 70B 93.4% on DeepInfra, 69.5% on SambaNova

来源

$0.09 to $5.63 per 1,000 correct calls

来源

Mikhail Maslenik ran our 800 requests on his own machine and scored 90.6% vs our 91.0%

来源