Can a local LLM replace Claude Code? The benchmark everyone quotes is the wrong one.

Description and use

The article explains that high SWE‑bench scores do not guarantee reliable agent behavior for local LLMs; key metrics are tool‑calling reliability and long‑horizon consistency. It recommends Qwen 3.6 27B on a 24 GB GPU.

Problem

Local LLMs often claim to match Claude Code on SWE‑bench scores, but they fail to reliably act as agents that emit valid tool calls, maintain long‑horizon consistency, handle large context, and follow instructions such as “don’t touch the tests.”

Outcome

The article clarifies that high SWE‑bench scores do not guarantee reliable agent behavior; tool‑calling reliability and long‑horizon consistency are the decisive factors for local LLMs to replace Claude Code. The recommended sweet spot is a single 24GB GPU running Qwen 3.6 27B.

Workflow

• 1. Identify the target use‑case (e.g., multi‑file editing, long‑horizon agentic work).
• 2. Review the 7 model families and their strengths/weaknesses (Qwen, GLM‑4, DeepSeek, Devstral, Kimi, GPT‑OSS, Gemma).
• 3. Check the model’s SWE‑bench Verified score and note the VRAM requirement.
• 4. Verify tool‑calling reliability by testing a sequence of at least 10 tool calls.
• 5. Test long‑horizon consistency by running a multi‑step task and ensuring the plan is preserved.
• 6. Evaluate context handling by feeding ~40K tokens of repository data.
• 7. Confirm instruction following (e.g., “don’t touch tests”) in a controlled scenario.
• 8. Compare results against the cost model and GPU efficiency curve to choose the sweet spot (single 24GB GPU with Qwen 3.6 27B).

Evidence

SWE‑bench measures "given a bug, produce a correct patch." That's a writing task.

Source

Tool‑calling is a separate trained skill from writing code.

Source

Reliability falls off a cliff below ~4B parameters.

Source

A single 24GB consumer GPU running Qwen 3.6 27B is the sweet spot for self‑hosting.

Source