I built a pip tool to test how well any local LLM calls tools (here is what my demo run showed)
Описание и применение
ToolCallBench – это инструмент для проверки способности локальных LLM выполнять вызовы файловых инструментов на MacBook M4 с 16 ГБ памяти. В демонстрационном запуске модели gemma4:e4b показали 95 % успешных задач, в то время как qwen3:8b, llama3.1:8b и qwen3.5:0.8b получили 75 %, 60 % и 55 % соответственно. Тесты делятся на чтение, редактирование, запуск, цепочки и ошибки, а оценка основана на корректности вызовов и конечном состоянии репозитория.
Задача
Determine which local LLM can reliably perform file‑system tool calls on a MacBook M4 with 16 GB RAM.
Результат
The demo run on a 16 GB MacBook M4 showed gemma4:e4b achieving 95 % overall pass, while qwen3:8b, llama3.1:8b, and qwen3.5:0.8b scored 75 %, 60 %, and 55 % respectively. Edit tasks were the most discriminating group, and chain tasks revealed weaknesses in multi‑step reasoning.
Рабочий процесс
- Install ToolCallBench via pip
- Create configs/ folder with model names in configs/my-models.yaml
- Run
toolcallbench doctor --probeto verify server connectivity - Execute
toolcallbench run --config configs/my-models.yamlto run the benchmark - Generate a report with
toolcallbench report
Доказательства
ToolCallBench tests tool calling on your own models and hardware.
I wanted a simple answer to a simple question: which local model can I trust to change files on my own machine?
The model gets six tools: read a file, list a folder, search, edit, apply a patch, and run a command.
A run passes only if every tool call was valid, the right tools were used, and the repository ends in the expected state.
I ran four models I already had: 20 tasks each, one run per task, Ollama 0.32.15 on a MacBook M4 with 16GB.
pip install toolcallbench toolcallbench init
toolcallbench doctor --probe toolcallbench run --config configs/my-models.yaml toolcallbench report
- Дата записи: 2026-10-24T12:11:36.327Z