How I made a 20B model code like a frontier one — by fixing the harness, not the model

Description and use

The author built the Ajna terminal coding agent and, by adding a repair layer to the harness, enabled a 20B model to perform coding tasks with frontier‑model reliability, cutting task time from 81 s to 15 s and improving success rates.

Problem

Small free models drift when calling tools, inventing names, passing wrong arguments, corrupting files, and failing to recover from errors, leading to wasted turns and unreliable behavior.

Outcome

The harness layer enabled a 20B model to perform coding tasks with reliability comparable to a frontier model, reducing task time from 81 s to 15 s and improving success rates across models.

Workflow

• Build Ajna, a pure‑Python coding agent with no third‑party packages.
• Implement a repair layer that maps invented tool names to real tools.
• Alias and coerce arguments to match tool schemas, dropping unknown keys.
• Reject replacement text that looks like a diff before writing to a file.
• Parse Python, JSON, XML, and TOML after each write and return an error if unparseable.
• Detect and report deletions of existing functions or classes.
• Fallback to literal search when regex is invalid.
• Benchmark the agent on ten tasks scored by running code, measuring success rates and run‑to‑run reliability.
• Iterate on harness checks to reduce reliance on model prompts for verification.
• Ship the tool with a public benchmark repository for community feedback.

Evidence

A coding agent is a model plus a harness. We spend all our attention on the model. I spent mine on the harness — and a free 20B model started finishing tasks that used to need a frontier one.

Source

The measured effect on one task: a failing-test fix that took gpt-oss:120b 81 seconds and left a corrupted file before this layer existed now completes on gpt-oss:20b — the smallest model available — in 15 seconds, correctly.

Source

The obvious way to make an agent more reliable is to tell the model to double-check itself: "when you're done, go and verify your work." I added exactly that and benchmarked it. On the bigger model it helped. On the smaller one it was a disaster:…

Source

Ten tasks, each run in a fresh temp directory, each scored by running code rather than by reading what the model said: fix a bug the tests catch, wire a feature end to end, rename across files, edit JSON precisely, remove duplication without changing behaviour, repair a file that doesn't parse — and, crucially, answer a question without touching anything.

Source