Most AI Second Opinions Are Fake. I Built a Two-LLM Review Engine to Prove It.

Описание и применение

Большинство вторых мнений ИИ не являются независимыми, что приводит к предвзятым обзорам. Созданный автором AdversarialDebate двухмодельный движок обеспечивает независимый анализ и структурированный дебат, повышая качество обзоров PR в реальных проектах.

Задача

AI second opinions often lack true independence, leading to biased reviews that simply echo the first model’s framing and conclusions.

Результат

Field tests on 70 real PRs from major open-source projects showed that independent, adversarial reviews produced 152 verdicts and 259 disputed outcomes, with 49/49 PRs having at least one claim matching ground truth. Model pairing significantly affected convergence, with DeepSeek+Mistral achieving 97% verdict rate versus 4% for GPT+Gemini.

Рабочий процесс

• Normalize PR diff into reviewable artifact
• Reviewer A analyzes artifact in isolation
• Reviewer B analyzes same artifact in isolation
• Persist both reviews separately
• Open revelation gate after both commits
• Conduct bounded debate rounds responding to each other’s claims
• Produce joint verdict or disagreement report

Доказательства

49 of 49 PRs with documented outcomes had at least one debate claim that matched the actual cause

Источник

152/411 debates reached verdict, 259/411 stayed disputed

Источник

DeepSeek + Mistral 97% verdict rate vs GPT + Gemini 4%

Источник