Can Local LLMs Actually Call Functions? I Tested 2 Models on 6 Real API Tasks

Описание и применение

Проверка локальных LLM на вызов функций показала, что Llama‑3.2‑3B быстрее и генерирует более корректный JSON, но Qwen3‑14B лучше справляется с точностью аргументов, многократными взаимодействиями и восстановлением после ошибок.

Задача

Local LLMs need to reliably produce valid JSON tool calls with correct arguments for real-world APIs, but existing claims are unverified.

Результат

Llama‑3.2‑3B: 98% JSON validity, 24 tok/s, 0.4 s first‑token, 4.1 GB RAM – best for high‑speed, low‑cost agents. Qwen3‑14B: 95% JSON validity, 5 tok/s, 1.2 s first‑token, 9.3 GB RAM – best for multi‑turn accuracy and error recovery.

Рабочий процесс

• 1. Clone benchmark repo
• 2. Install dependencies and pull models via Ollama
• 3. Run benchmark with --models all
• 4. Review JSON results and agent traces

Доказательства

Both models cleared 95% on basic JSON validity. Llama‑3.2‑3B actually edged out Qwen3‑14B here — the smaller 3B model has a tighter tool‑calling format that's more reliable in practice.

Источник

Qwen3‑14B extracts the right values from natural language 82% of the time vs 71% for the 3B model. On multi‑argument tasks Qwen3 wins by 11 points. On multi‑turn context Qwen3 wins by 20 points.

Источник

Qwen3 recovers correctly 65% of the time. Llama‑3.2‑3B recovers 48% of the time and hallucinates a "fix" 31% of the time.

Источник