Most AI Second Opinions Are Fake. I Built a Two-LLM Review Engine to Prove It.

说明与用途

大多数 AI 二次意见缺乏真正的独立性，导致评审偏见。作者构建了 AdversarialDebate 两模型引擎，强制独立分析和结构化辩论，提升真实开源项目的 PR 评审质量。

问题

AI second opinions often lack true independence, leading to biased reviews that simply echo the first model’s framing and conclusions.

结果

Field tests on 70 real PRs from major open-source projects showed that independent, adversarial reviews produced 152 verdicts and 259 disputed outcomes, with 49/49 PRs having at least one claim matching ground truth. Model pairing significantly affected convergence, with DeepSeek+Mistral achieving 97% verdict rate versus 4% for GPT+Gemini.

工作流

• Normalize PR diff into reviewable artifact
• Reviewer A analyzes artifact in isolation
• Reviewer B analyzes same artifact in isolation
• Persist both reviews separately
• Open revelation gate after both commits
• Conduct bounded debate rounds responding to each other’s claims
• Produce joint verdict or disagreement report

证据

49 of 49 PRs with documented outcomes had at least one debate claim that matched the actual cause

来源

152/411 debates reached verdict, 259/411 stayed disputed

来源

DeepSeek + Mistral 97% verdict rate vs GPT + Gemini 4%

来源