LLM-as-Judge Position Bias: Swapping A/B Flipped 87 of 300 Verdicts
说明与用途
LLM评判者的位置信息偏差导致首位答案被偏好。解决方案:每对答案反向评估两次,仅在两次评估一致时计为胜利。
问题
Pairwise LLM judges prefer the answer that appears first in the prompt, skewing evaluation results.
结果
Running each pair twice and only counting consistent wins removes position bias, yielding a fairer comparison of new vs old answers.
工作流
- 1. Prepare question and two answer texts.
- 2. Run judge with new answer in slot A and old in slot B.
- 3. Run judge with old answer in slot A and new in slot B.
- 4. If run1=="A" and run2=="B", record "new" win; if run1=="B" and run2=="A", record "old" win; otherwise record "tie".
- 5. Aggregate results: count new wins, old wins, and ties.
- 6. Report percentages of new wins, old wins, and ties.
- 7. Optionally perform a mirror test with identical answers to detect bias.
证据
The new prompt won 43% when swapped
Count a win only when both runs agree, and treat disagreement as a tie
Run the mirror test: put the identical answer in both slots
- 记录日期: 2026-11-01T22:44:13.874Z