Tīrtha Bench: the model data coders actually need (and what we're building next)

Description and use

Tīrtha Bench evaluates 68 models on 800 tool‑calling tasks, reporting accuracy, cost per correct answer, speed, and host effects. Results are publicly available on tirtha.ai/bench, with per‑model pages and raw data.

Problem

Developers lack a single source to compare language models on tool‑calling performance, cost, speed, and host variability.

Outcome

The benchmark shows significant host‑dependent performance (e.g., Llama 3.3 70B: 93.4% vs 69.5%) and wide cost‑per‑correct‑answer ranges ($0.09–$5.63 per 1,000 correct calls).

Workflow

• 1. Collect 68 models. 2. Run each on 800 tool‑calling tasks (400 single, 200 multi‑tool, 200 parallel) plus 120 trap prompts. 3. Score all runs with a consistent checker. 4. Compute accuracy, cost per 1,000 correct calls, speed, and trap success. 5. Publish results on tirtha.ai/bench with per‑model pages and raw data.
• 6. Allow external users to reproduce runs and compare results.

Evidence

We could not find that answered in one place, so we built it.

Source

68 models tested, 8 on the same shared test, 800 tasks and 120 traps, same scorer

Source

Same host matters: Llama 3.3 70B 93.4% on DeepInfra, 69.5% on SambaNova

Source

$0.09 to $5.63 per 1,000 correct calls

Source

Mikhail Maslenik ran our 800 requests on his own machine and scored 90.6% vs our 91.0%

Source