Can Local LLMs Actually Call Functions? I Tested 2 Models on 6 Real API Tasks

说明与用途

本次基准测试表明，Llama‑3.2‑3B 在速度和 JSON 合规性方面表现更佳，而 Qwen3‑14B 在参数准确性、多轮交互和错误恢复方面更胜一筹，为本地 LLM 代码代理的模型选择提供依据。

问题

Local LLMs need to reliably produce valid JSON tool calls with correct arguments for real-world APIs, but existing claims are unverified.

结果

Llama‑3.2‑3B: 98% JSON validity, 24 tok/s, 0.4 s first‑token, 4.1 GB RAM – best for high‑speed, low‑cost agents. Qwen3‑14B: 95% JSON validity, 5 tok/s, 1.2 s first‑token, 9.3 GB RAM – best for multi‑turn accuracy and error recovery.

工作流

• 1. Clone benchmark repo
• 2. Install dependencies and pull models via Ollama
• 3. Run benchmark with --models all
• 4. Review JSON results and agent traces

证据

Both models cleared 95% on basic JSON validity. Llama‑3.2‑3B actually edged out Qwen3‑14B here — the smaller 3B model has a tighter tool‑calling format that's more reliable in practice.

来源

Qwen3‑14B extracts the right values from natural language 82% of the time vs 71% for the 3B model. On multi‑argument tasks Qwen3 wins by 11 points. On multi‑turn context Qwen3 wins by 20 points.

来源

Qwen3 recovers correctly 65% of the time. Llama‑3.2‑3B recovers 48% of the time and hallucinates a "fix" 31% of the time.

来源