I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy

说明与用途

在隔离的 VM 中评估的 Harness Engineering 模式可将令牌减少至 95% 并实现 100% 的一次性通过率，提升 LLM 代理的速度和成本效率。

问题

AI coding agents often generate excessive tokens and fail to pass tests in a single turn, leading to high costs and unreliable deployments.

结果

The benchmark identified seven high‑impact harness patterns that cut input tokens by 90–95%, reduced API costs by 70–84%, and improved speed by 1.5–2.3×, achieving 100% first‑turn pass rates on a hidden 20‑case suite.

工作流

• 1. Deploy a disposable Debian 13 Proxmox VM with no network interfaces.
• 2. Run the agent harness pattern experiment (E04–E44) inside the VM.
• 3. Feed identical initial codebases and hidden 20‑case grading suite to the agent.
• 4. Measure input tokens, API cost, speed, and accuracy (first‑turn pass rate).
• 5. Compare results across patterns to identify top performers.

证据

All 41 A/B experiments were executed inside disposable Debian 13 Proxmox VMs with zero network interfaces (0 NICs)

来源

Episodic Gotchas Memory (E20): Injecting 5 repo‑specific gotchas reduced input tokens by 95.5% and achieved a 1‑turn 20/20 pass

来源

Tree‑sitter Skeleton Context (E21): Passing only function signatures & type contracts hit 20/20 in 1 turn

来源