I built a hackathon judging app and it was too confident

Описание и применение

Приложение для оценки хакатона использовало модель с аддитивным смещением и резэмплингом оценок, но оказалось слишком уверенным из‑за ограниченного количества обзоров. Были выявлены судьи, которые по одиночке могут изменить топ‑5.

Задача

The app produced overly confident ranking results because the resampling method only shuffled existing reviews and did not account for judge variability, leading to inaccurate uncertainty estimates.

Результат

The additive model outperformed plain average and z‑score methods (Spearman 0.813 vs 0.697 and 0.778). Resampling provided useful uncertainty signals, though it was overconfident due to limited review variability. The kingmaker check revealed that 9 of 30 judges could single‑handedly change top‑5 placement, highlighting the need for judge‑level uncertainty modeling.

Рабочий процесс

• Collect raw score sheets from organizers
• Detect and exclude flat‑liner judges
• Fit additive model iteratively for quality and leniency with ridge regularisation
• Resample each project's reviews 300 times, refit model each time, and count top‑5 occurrences
• Compute chance of top‑5 finish for each project
• Flag close calls (20–80% chance)
• Suggest additional judges for close‑call projects
• Run kingmaker check by refitting model excluding each judge in turn and flagging changes in top‑5 or first place

Доказательства

The flat‑lining judge has a spread of zero. So does any judge who only reviewed one project. Divide by zero, and those projects can't be ranked at all.

Источник

It never divides by anyone's spread, so the flat‑liner can't break it.

Источник

The resampling only shuffles the reviews a project already has. With two or three reviews there just aren't many ways to shuffle, so the spread looks small.

Источник

9 of 30 judges each move a top‑5 place on their own. And two of them flip first place, the one place the resampling called fairly safe.

Источник