I built a pip tool to test how well any local LLM calls tools (here is what my demo run showed)

Description and use

ToolCallBench is a benchmark for local LLMs to test file‑system tool calls on a MacBook M4 with 16 GB RAM. In a demo run, gemma4:e4b achieved 95 % pass, while qwen3:8b, llama3.1:8b, and qwen3.5:0.8b scored 75 %, 60 %, and 55 %. Tasks cover read, edit, run, chain, and failure scenarios, with scoring based on valid calls and final repo state.

Problem

Determine which local LLM can reliably perform file‑system tool calls on a MacBook M4 with 16 GB RAM.

Outcome

The demo run on a 16 GB MacBook M4 showed gemma4:e4b achieving 95 % overall pass, while qwen3:8b, llama3.1:8b, and qwen3.5:0.8b scored 75 %, 60 %, and 55 % respectively. Edit tasks were the most discriminating group, and chain tasks revealed weaknesses in multi‑step reasoning.

Workflow

• Install ToolCallBench via pip
• Create configs/ folder with model names in configs/my-models.yaml
• Run toolcallbench doctor --probe to verify server connectivity
• Execute toolcallbench run --config configs/my-models.yaml to run the benchmark
• Generate a report with toolcallbench report

Evidence

ToolCallBench tests tool calling on your own models and hardware.

Source

I wanted a simple answer to a simple question: which local model can I trust to change files on my own machine?

Source

The model gets six tools: read a file, list a folder, search, edit, apply a patch, and run a command.

Source

A run passes only if every tool call was valid, the right tools were used, and the repository ends in the expected state.

Source

I ran four models I already had: 20 tasks each, one run per task, Ollama 0.32.15 on a MacBook M4 with 16GB.

Source

pip install toolcallbench toolcallbench init

Source

toolcallbench doctor --probe toolcallbench run --config configs/my-models.yaml toolcallbench report

Source