Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

I built a pip tool to test how well any local LLM calls tools (here is what my demo run showed)

说明与用途

ToolCallBench 是一个用于在配备 16GB 内存的 MacBook M4 上测试本地 LLM 工具调用的基准工具。演示运行中,gemma4:e4b 的通过率为 95%,qwen3:8b、llama3.1:8b 和 qwen3.5:0.8b 分别为 75%、60% 和 55%。任务涵盖读取、编辑、运行、链式操作和失败场景,评分基于调用有效性和最终仓库状态。

问题

Determine which local LLM can reliably perform file‑system tool calls on a MacBook M4 with 16 GB RAM.

结果

The demo run on a 16 GB MacBook M4 showed gemma4:e4b achieving 95 % overall pass, while qwen3:8b, llama3.1:8b, and qwen3.5:0.8b scored 75 %, 60 %, and 55 % respectively. Edit tasks were the most discriminating group, and chain tasks revealed weaknesses in multi‑step reasoning.

工作流

  • Install ToolCallBench via pip
  • Create configs/ folder with model names in configs/my-models.yaml
  • Run toolcallbench doctor --probe to verify server connectivity
  • Execute toolcallbench run --config configs/my-models.yaml to run the benchmark
  • Generate a report with toolcallbench report

证据

ToolCallBench tests tool calling on your own models and hardware.

来源

I wanted a simple answer to a simple question: which local model can I trust to change files on my own machine?

来源

The model gets six tools: read a file, list a folder, search, edit, apply a patch, and run a command.

来源

A run passes only if every tool call was valid, the right tools were used, and the repository ends in the expected state.

来源

I ran four models I already had: 20 tasks each, one run per task, Ollama 0.32.15 on a MacBook M4 with 16GB.

来源

pip install toolcallbench toolcallbench init

来源

toolcallbench doctor --probe toolcallbench run --config configs/my-models.yaml toolcallbench report

来源

  • 记录日期: 2026-10-24T12:11:36.327Z