LLM-as-a-Judge: Claude Gave 68% of Answers a 7 or 8 Out of 10
说明与用途
Preterview 的 LLM 评判系统最初在 1–10 评分尺度上聚集在 7–8 分,降低了可靠性。改用带证据的二进制检查表将一致性提升至 0.79,降低了重复评分波动,并消除了分数堆积,但成本上升且需要处理简短但强有力的答案。
问题
The LLM-as-a-judge system produced clustered scores around 7–8 on a 1–10 scale, reducing grading reliability and consistency.
结果
Switching from a numeric prompt to a binary checklist with evidence requirement improved ranking agreement from 0.41 to 0.79, reduced rerun noise, and eliminated score clustering, though it increased cost and required handling short, strong answers.
工作流
- 1. Build a 60‑item hand‑graded calibration set on a 1–5 scale.
- 2. Measure baseline prompt performance (Spearman, mode share, rerun std).
- 3. Test fixes: (a) “be strict” instruction, (b) 1–5 scale with anchored levels, (c) add reference answers, (d) binary checklist with evidence requirement.
- 4. Evaluate each fix on the calibration set.
- 5. Adopt checklist approach, compute scores in code, enforce evidence quotes.
- 6. Deploy updated prompt to production.
- 7. Monitor cost increase (~75% per session) and short‑answer mis‑scoring.
- 8. Add patch for brief but complete answers.
证据
41% were a 7. 27% were an 8. Anything below 5 showed up 3% of the time.
Spearman ρ against my grades (does it rank answers correctly?)
The winner: ρ = 0.79, rerun std dev 0.31, and no pile-up at any single level.
The cost: scoring got ~75% more expensive per session, and the checklist still mis-scores short, unusual answers.
- 记录日期: 2026-10-09T17:41:12.584Z