Can a local LLM replace Claude Code? The benchmark everyone quotes is the wrong one.

说明与用途

文章指出，高 SWE‑bench 分数并不能保证本地 LLM 作为代理的可靠性；关键指标是工具调用可靠性和长期一致性。建议在 24 GB GPU 上使用 Qwen 3.6 27B。

问题

Local LLMs often claim to match Claude Code on SWE‑bench scores, but they fail to reliably act as agents that emit valid tool calls, maintain long‑horizon consistency, handle large context, and follow instructions such as “don’t touch the tests.”

结果

The article clarifies that high SWE‑bench scores do not guarantee reliable agent behavior; tool‑calling reliability and long‑horizon consistency are the decisive factors for local LLMs to replace Claude Code. The recommended sweet spot is a single 24GB GPU running Qwen 3.6 27B.

工作流

• 1. Identify the target use‑case (e.g., multi‑file editing, long‑horizon agentic work).
• 2. Review the 7 model families and their strengths/weaknesses (Qwen, GLM‑4, DeepSeek, Devstral, Kimi, GPT‑OSS, Gemma).
• 3. Check the model’s SWE‑bench Verified score and note the VRAM requirement.
• 4. Verify tool‑calling reliability by testing a sequence of at least 10 tool calls.
• 5. Test long‑horizon consistency by running a multi‑step task and ensuring the plan is preserved.
• 6. Evaluate context handling by feeding ~40K tokens of repository data.
• 7. Confirm instruction following (e.g., “don’t touch tests”) in a controlled scenario.
• 8. Compare results against the cost model and GPU efficiency curve to choose the sweet spot (single 24GB GPU with Qwen 3.6 27B).

证据

SWE‑bench measures "given a bug, produce a correct patch." That's a writing task.

来源

Tool‑calling is a separate trained skill from writing code.

来源

Reliability falls off a cliff below ~4B parameters.

来源

A single 24GB consumer GPU running Qwen 3.6 27B is the sweet spot for self‑hosting.

来源