Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

LLM-as-a-Judge: Claude Gave 68% of Answers a 7 or 8 Out of 10

Description and use

Preterview’s LLM judge originally clustered scores at 7–8 on a 1–10 scale, reducing reliability. Switching to a binary checklist with evidence raised agreement to 0.79, lowered rerun noise, and eliminated score pile‑ups, though it increased cost and required handling brief strong answers.

Problem

The LLM-as-a-judge system produced clustered scores around 7–8 on a 1–10 scale, reducing grading reliability and consistency.

Outcome

Switching from a numeric prompt to a binary checklist with evidence requirement improved ranking agreement from 0.41 to 0.79, reduced rerun noise, and eliminated score clustering, though it increased cost and required handling short, strong answers.

Workflow

  • 1. Build a 60‑item hand‑graded calibration set on a 1–5 scale.
  • 2. Measure baseline prompt performance (Spearman, mode share, rerun std).
  • 3. Test fixes: (a) “be strict” instruction, (b) 1–5 scale with anchored levels, (c) add reference answers, (d) binary checklist with evidence requirement.
  • 4. Evaluate each fix on the calibration set.
  • 5. Adopt checklist approach, compute scores in code, enforce evidence quotes.
  • 6. Deploy updated prompt to production.
  • 7. Monitor cost increase (~75% per session) and short‑answer mis‑scoring.
  • 8. Add patch for brief but complete answers.

Evidence

41% were a 7. 27% were an 8. Anything below 5 showed up 3% of the time.

Source

Spearman ρ against my grades (does it rank answers correctly?)

Source

The winner: ρ = 0.79, rerun std dev 0.31, and no pile-up at any single level.

Source

The cost: scoring got ~75% more expensive per session, and the checklist still mis-scores short, unusual answers.

Source

  • Record date: 2026-10-09T17:41:12.584Z