Searching for Orbi's harness, part 1: what our reviewer missed

Description and use

The article explains how the team refined Orbi’s harness to eliminate regressions and improve automated code review accuracy. After iterative tuning of prompts, skills, and models, they achieved 9/9 successful tests on both tuned and unseen issues, though token usage and runtime increased.

Problem

Orbi's automated code review loop sometimes fails to detect regressions or introduces new bugs due to shortcomings in its harness configuration.

Outcome

The tuned harness (v15) with deepseek‑flash for both roles achieved 9/9 pass on tuned issues and unseen bugs, while the original harness failed many. Token usage increased 2× and wall‑clock time 1.45×, but no measurable quality gain on ordinary bugs.

Workflow

• 1. Identify failures in Orbi‑generated PRs.
• 2. Run Orbi’s full loop on tuned issues, unseen bugs, and regression‑prone bugs.
• 3. Swap prompts, skills, tools, implementation and review models.
• 4. Grade each merged result with a hidden grader unseen by Orbi.
• 5. Analyze failures, adjust harness, repeat until desired pass rate.
• 6. Benchmark harness versions against model setups.
• 7. Publish results and scripts on benchmark page and GitHub.

Evidence

On the issues we tuned on, the harness moved the numbers more than any model swap we tried.

Source

With deepseek-flash in both roles, the original harness passed 6 of 15 runs on the three tuning issues. v13 to v15 passed 28 of 30.

Source

On nine unseen issues, the original harness, v15 and v15 with a sol reviewer all passed 9 of 9.

Source

The grader was calibrated to fail the original code, fail Orbi's merged fork PR, and pass a hand‑verified fix.

Source