LLM-as-a-Judge: Claude Gave 68% of Answers a 7 or 8 Out of 10
Описание и применение
Система оценки кандидатов на основе LLM в Preterview изначально выдавалась сконцентрированными оценками 7–8 из 10, что снижало точность. Перевод на двоичный чеклист с доказательствами повысил согласованность с ручной оценкой до 0,79, уменьшил разброс повторных оценок и устранил кластеризацию, но увеличил стоимость и потребовал дополнительной обработки коротких ответов.
Задача
The LLM-as-a-judge system produced clustered scores around 7–8 on a 1–10 scale, reducing grading reliability and consistency.
Результат
Switching from a numeric prompt to a binary checklist with evidence requirement improved ranking agreement from 0.41 to 0.79, reduced rerun noise, and eliminated score clustering, though it increased cost and required handling short, strong answers.
Рабочий процесс
- 1. Build a 60‑item hand‑graded calibration set on a 1–5 scale.
- 2. Measure baseline prompt performance (Spearman, mode share, rerun std).
- 3. Test fixes: (a) “be strict” instruction, (b) 1–5 scale with anchored levels, (c) add reference answers, (d) binary checklist with evidence requirement.
- 4. Evaluate each fix on the calibration set.
- 5. Adopt checklist approach, compute scores in code, enforce evidence quotes.
- 6. Deploy updated prompt to production.
- 7. Monitor cost increase (~75% per session) and short‑answer mis‑scoring.
- 8. Add patch for brief but complete answers.
Доказательства
41% were a 7. 27% were an 8. Anything below 5 showed up 3% of the time.
Spearman ρ against my grades (does it rank answers correctly?)
The winner: ρ = 0.79, rerun std dev 0.31, and no pile-up at any single level.
The cost: scoring got ~75% more expensive per session, and the checklist still mis-scores short, unusual answers.
- Дата записи: 2026-10-09T17:41:12.584Z