Tīrtha Bench: the model data coders actually need (and what we're building next)

说明与用途

Tīrtha Bench 是唯一比较模型在工具调用准确性、速度、成本和主机影响的平台。它在 800 个任务加 120 个陷阱上测试 68 个模型，统一评分。结果公开且可外部复现。

问题

Developers lack a single source to compare model performance on tool‑calling tasks, including accuracy, speed, cost, and host effects.

结果

The benchmark shows significant variation in accuracy and cost per correct answer across models and hosts, with a 65× cost spread and host‑dependent performance differences.

工作流

• 1. Collect 800 tool‑calling tasks and 120 trap prompts.
• 2. Run each model on the same test set with identical prompts.
• 3. Score results using a fixed rule set and Berkeley’s checker.
• 4. Publish raw results and per‑model pages on tirtha.ai/bench.
• 5. Allow external reproducibility by publishing requests and enabling others to run the same benchmark.

证据

We could not find that answered in one place, so we built it.

来源

68 models tested, 8 on the same shared test, 800 tasks and 120 traps, same scorer

来源

The host matters, not just the model. The same open model, gpt-oss-120b, on the same 800 tasks, scored anywhere from 63.9% to 78.0% depending on which of four providers served it.

来源

A cheap model that gets more wrong can cost more per correct call. So we divide what each run actually cost by the answers it got right.

来源

Mikhail Maslenik ran our 800 requests on his own machine, on a quantized copy of the same model, and scored them with Berkeley's official checker. He got 90.6%. We got 91.0%.

来源