Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

LLM-as-Judge Position Bias: Swapping A/B Flipped 87 of 300 Verdicts

说明与用途

LLM评判者的位置信息偏差导致首位答案被偏好。解决方案:每对答案反向评估两次,仅在两次评估一致时计为胜利。

问题

Pairwise LLM judges prefer the answer that appears first in the prompt, skewing evaluation results.

结果

Running each pair twice and only counting consistent wins removes position bias, yielding a fairer comparison of new vs old answers.

工作流

  • 1. Prepare question and two answer texts.
  • 2. Run judge with new answer in slot A and old in slot B.
  • 3. Run judge with old answer in slot A and new in slot B.
  • 4. If run1=="A" and run2=="B", record "new" win; if run1=="B" and run2=="A", record "old" win; otherwise record "tie".
  • 5. Aggregate results: count new wins, old wins, and ties.
  • 6. Report percentages of new wins, old wins, and ties.
  • 7. Optionally perform a mirror test with identical answers to detect bias.

证据

The new prompt won 43% when swapped

来源

Count a win only when both runs agree, and treat disagreement as a tie

来源

Run the mirror test: put the identical answer in both slots

来源

  • 记录日期: 2026-11-01T22:44:13.874Z