LLM-as-Judge Position Bias: Swapping A/B Flipped 87 of 300 Verdicts

Description and use

Position bias in LLM judges favors the answer that appears first in the prompt. Fix: evaluate each pair twice in opposite orders and count a win only when both runs agree.

Problem

Pairwise LLM judges prefer the answer that appears first in the prompt, skewing evaluation results.

Outcome

Running each pair twice and only counting consistent wins removes position bias, yielding a fairer comparison of new vs old answers.

Workflow

• 1. Prepare question and two answer texts.
• 2. Run judge with new answer in slot A and old in slot B.
• 3. Run judge with old answer in slot A and new in slot B.
• 4. If run1=="A" and run2=="B", record "new" win; if run1=="B" and run2=="A", record "old" win; otherwise record "tie".
• 5. Aggregate results: count new wins, old wins, and ties.
• 6. Report percentages of new wins, old wins, and ties.
• 7. Optionally perform a mirror test with identical answers to detect bias.

Evidence

The new prompt won 43% when swapped

Source

Count a win only when both runs agree, and treat disagreement as a tie

Source

Run the mirror test: put the identical answer in both slots

Source