Searching for Orbi's harness, part 1: what our reviewer missed
说明与用途
文章说明团队如何改进 Orbi 的 harness,以消除回归并提高自动代码审查的准确性。通过对 prompts、技能和模型的迭代调优,他们在已调试和未见过的问题上都实现了 9/9 的成功测试,但代币使用量和运行时间有所增加。
问题
Orbi's automated code review loop sometimes fails to detect regressions or introduces new bugs due to shortcomings in its harness configuration.
结果
The tuned harness (v15) with deepseek‑flash for both roles achieved 9/9 pass on tuned issues and unseen bugs, while the original harness failed many. Token usage increased 2× and wall‑clock time 1.45×, but no measurable quality gain on ordinary bugs.
工作流
- 1. Identify failures in Orbi‑generated PRs.
- 2. Run Orbi’s full loop on tuned issues, unseen bugs, and regression‑prone bugs.
- 3. Swap prompts, skills, tools, implementation and review models.
- 4. Grade each merged result with a hidden grader unseen by Orbi.
- 5. Analyze failures, adjust harness, repeat until desired pass rate.
- 6. Benchmark harness versions against model setups.
- 7. Publish results and scripts on benchmark page and GitHub.
证据
On the issues we tuned on, the harness moved the numbers more than any model swap we tried.
With deepseek-flash in both roles, the original harness passed 6 of 15 runs on the three tuning issues. v13 to v15 passed 28 of 30.
On nine unseen issues, the original harness, v15 and v15 with a sol reviewer all passed 9 of 9.
The grader was calibrated to fail the original code, fail Orbi's merged fork PR, and pass a hand‑verified fix.
- 记录日期: 2026-10-05T16:41:00.717Z