Can a local LLM replace Claude Code? The benchmark everyone quotes is the wrong one.

Описание и применение

В статье объясняется, что высокие баллы SWE‑bench не гарантируют надёжного поведения локальных LLM как агентов; ключевыми являются надёжность вызовов инструментов и согласованность в долгосрочных задачах. Рекомендуется использовать Qwen 3.6 27B на видеокарте 24 GB.

Задача

Local LLMs often claim to match Claude Code on SWE‑bench scores, but they fail to reliably act as agents that emit valid tool calls, maintain long‑horizon consistency, handle large context, and follow instructions such as “don’t touch the tests.”

Результат

The article clarifies that high SWE‑bench scores do not guarantee reliable agent behavior; tool‑calling reliability and long‑horizon consistency are the decisive factors for local LLMs to replace Claude Code. The recommended sweet spot is a single 24GB GPU running Qwen 3.6 27B.

Рабочий процесс

• 1. Identify the target use‑case (e.g., multi‑file editing, long‑horizon agentic work).
• 2. Review the 7 model families and their strengths/weaknesses (Qwen, GLM‑4, DeepSeek, Devstral, Kimi, GPT‑OSS, Gemma).
• 3. Check the model’s SWE‑bench Verified score and note the VRAM requirement.
• 4. Verify tool‑calling reliability by testing a sequence of at least 10 tool calls.
• 5. Test long‑horizon consistency by running a multi‑step task and ensuring the plan is preserved.
• 6. Evaluate context handling by feeding ~40K tokens of repository data.
• 7. Confirm instruction following (e.g., “don’t touch tests”) in a controlled scenario.
• 8. Compare results against the cost model and GPU efficiency curve to choose the sweet spot (single 24GB GPU with Qwen 3.6 27B).

Доказательства

SWE‑bench measures "given a bug, produce a correct patch." That's a writing task.

Источник

Tool‑calling is a separate trained skill from writing code.

Источник

Reliability falls off a cliff below ~4B parameters.

Источник

A single 24GB consumer GPU running Qwen 3.6 27B is the sweet spot for self‑hosting.

Источник