I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy

Description and use

Harness Engineering patterns evaluated in isolated VMs cut tokens by up to 95 % and achieve 100 % first‑turn pass rates, improving speed and cost efficiency of LLM agents.

Problem

AI coding agents often generate excessive tokens and fail to pass tests in a single turn, leading to high costs and unreliable deployments.

Outcome

The benchmark identified seven high‑impact harness patterns that cut input tokens by 90–95%, reduced API costs by 70–84%, and improved speed by 1.5–2.3×, achieving 100% first‑turn pass rates on a hidden 20‑case suite.

Workflow

• 1. Deploy a disposable Debian 13 Proxmox VM with no network interfaces.
• 2. Run the agent harness pattern experiment (E04–E44) inside the VM.
• 3. Feed identical initial codebases and hidden 20‑case grading suite to the agent.
• 4. Measure input tokens, API cost, speed, and accuracy (first‑turn pass rate).
• 5. Compare results across patterns to identify top performers.

Evidence

All 41 A/B experiments were executed inside disposable Debian 13 Proxmox VMs with zero network interfaces (0 NICs)

Source

Episodic Gotchas Memory (E20): Injecting 5 repo‑specific gotchas reduced input tokens by 95.5% and achieved a 1‑turn 20/20 pass

Source

Tree‑sitter Skeleton Context (E21): Passing only function signatures & type contracts hit 20/20 in 1 turn

Source