How I made a 20B model code like a frontier one — by fixing the harness, not the model

说明与用途

作者构建了 Ajna 终端编码代理，并通过在 harness 中添加修复层，使 20B 模型能够以前沿模型的可靠性执行编码任务，将任务时间从 81 秒缩短到 15 秒，并提高成功率。

问题

Small free models drift when calling tools, inventing names, passing wrong arguments, corrupting files, and failing to recover from errors, leading to wasted turns and unreliable behavior.

结果

The harness layer enabled a 20B model to perform coding tasks with reliability comparable to a frontier model, reducing task time from 81 s to 15 s and improving success rates across models.

工作流

• Build Ajna, a pure‑Python coding agent with no third‑party packages.
• Implement a repair layer that maps invented tool names to real tools.
• Alias and coerce arguments to match tool schemas, dropping unknown keys.
• Reject replacement text that looks like a diff before writing to a file.
• Parse Python, JSON, XML, and TOML after each write and return an error if unparseable.
• Detect and report deletions of existing functions or classes.
• Fallback to literal search when regex is invalid.
• Benchmark the agent on ten tasks scored by running code, measuring success rates and run‑to‑run reliability.
• Iterate on harness checks to reduce reliance on model prompts for verification.
• Ship the tool with a public benchmark repository for community feedback.

证据

A coding agent is a model plus a harness. We spend all our attention on the model. I spent mine on the harness — and a free 20B model started finishing tasks that used to need a frontier one.

来源

The measured effect on one task: a failing-test fix that took gpt-oss:120b 81 seconds and left a corrupted file before this layer existed now completes on gpt-oss:20b — the smallest model available — in 15 seconds, correctly.

来源

The obvious way to make an agent more reliable is to tell the model to double-check itself: "when you're done, go and verify your work." I added exactly that and benchmarked it. On the bigger model it helped. On the smaller one it was a disaster:…

来源

Ten tasks, each run in a fresh temp directory, each scored by running code rather than by reading what the model said: fix a bug the tests catch, wire a feature end to end, rename across files, edit JSON precisely, remove duplication without changing behaviour, repair a file that doesn't parse — and, crucially, answer a question without touching anything.

来源