Tīrtha Bench: the model data coders actually need (and what we're building next)

Description and use

Tīrtha Bench is the sole platform comparing models on tool‑calling accuracy, speed, cost, and host impact. It tests 68 models on 800 tasks plus 120 traps, scored uniformly. Results are publicly available and externally reproducible.

Problem

Developers lack a single source to compare model performance on tool‑calling tasks, including accuracy, speed, cost, and host effects.

Outcome

The benchmark shows significant variation in accuracy and cost per correct answer across models and hosts, with a 65× cost spread and host‑dependent performance differences.

Workflow

• 1. Collect 800 tool‑calling tasks and 120 trap prompts.
• 2. Run each model on the same test set with identical prompts.
• 3. Score results using a fixed rule set and Berkeley’s checker.
• 4. Publish raw results and per‑model pages on tirtha.ai/bench.
• 5. Allow external reproducibility by publishing requests and enabling others to run the same benchmark.

Evidence

We could not find that answered in one place, so we built it.

Source

68 models tested, 8 on the same shared test, 800 tasks and 120 traps, same scorer

Source

The host matters, not just the model. The same open model, gpt-oss-120b, on the same 800 tasks, scored anywhere from 63.9% to 78.0% depending on which of four providers served it.

Source

A cheap model that gets more wrong can cost more per correct call. So we divide what each run actually cost by the answers it got right.

Source

Mikhail Maslenik ran our 800 requests on his own machine, on a quantized copy of the same model, and scored them with Berkeley's official checker. He got 90.6%. We got 91.0%.

Source