Most AI Second Opinions Are Fake. I Built a Two-LLM Review Engine to Prove It.

Description and use

Most AI second opinions lack true independence, leading to biased reviews. The author built AdversarialDebate, a two-LLM engine that forces independent analysis and structured debate, improving pull‑request review quality on real open‑source projects.

Problem

AI second opinions often lack true independence, leading to biased reviews that simply echo the first model’s framing and conclusions.

Outcome

Field tests on 70 real PRs from major open-source projects showed that independent, adversarial reviews produced 152 verdicts and 259 disputed outcomes, with 49/49 PRs having at least one claim matching ground truth. Model pairing significantly affected convergence, with DeepSeek+Mistral achieving 97% verdict rate versus 4% for GPT+Gemini.

Workflow

• Normalize PR diff into reviewable artifact
• Reviewer A analyzes artifact in isolation
• Reviewer B analyzes same artifact in isolation
• Persist both reviews separately
• Open revelation gate after both commits
• Conduct bounded debate rounds responding to each other’s claims
• Produce joint verdict or disagreement report

Evidence

49 of 49 PRs with documented outcomes had at least one debate claim that matched the actual cause

Source

152/411 debates reached verdict, 259/411 stayed disputed

Source

DeepSeek + Mistral 97% verdict rate vs GPT + Gemini 4%

Source