I built a pip tool to test how well any local LLM calls tools (here is what my demo run showed)
说明与用途
ToolCallBench 是一个用于在配备 16GB 内存的 MacBook M4 上测试本地 LLM 工具调用的基准工具。演示运行中,gemma4:e4b 的通过率为 95%,qwen3:8b、llama3.1:8b 和 qwen3.5:0.8b 分别为 75%、60% 和 55%。任务涵盖读取、编辑、运行、链式操作和失败场景,评分基于调用有效性和最终仓库状态。
问题
Determine which local LLM can reliably perform file‑system tool calls on a MacBook M4 with 16 GB RAM.
结果
The demo run on a 16 GB MacBook M4 showed gemma4:e4b achieving 95 % overall pass, while qwen3:8b, llama3.1:8b, and qwen3.5:0.8b scored 75 %, 60 %, and 55 % respectively. Edit tasks were the most discriminating group, and chain tasks revealed weaknesses in multi‑step reasoning.
工作流
- Install ToolCallBench via pip
- Create configs/ folder with model names in configs/my-models.yaml
- Run
toolcallbench doctor --probeto verify server connectivity - Execute
toolcallbench run --config configs/my-models.yamlto run the benchmark - Generate a report with
toolcallbench report
证据
ToolCallBench tests tool calling on your own models and hardware.
I wanted a simple answer to a simple question: which local model can I trust to change files on my own machine?
The model gets six tools: read a file, list a folder, search, edit, apply a patch, and run a command.
A run passes only if every tool call was valid, the right tools were used, and the repository ends in the expected state.
I ran four models I already had: 20 tasks each, one run per task, Ollama 0.32.15 on a MacBook M4 with 16GB.
pip install toolcallbench toolcallbench init
toolcallbench doctor --probe toolcallbench run --config configs/my-models.yaml toolcallbench report
- 记录日期: 2026-10-24T12:11:36.327Z