I built a hackathon judging app and it was too confident

说明与用途

该黑客松评审应用使用加法模型和重采样，但由于评审数量有限，导致过度自信。识别出单独能改变前五名的评审。

问题

The app produced overly confident ranking results because the resampling method only shuffled existing reviews and did not account for judge variability, leading to inaccurate uncertainty estimates.

结果

The additive model outperformed plain average and z‑score methods (Spearman 0.813 vs 0.697 and 0.778). Resampling provided useful uncertainty signals, though it was overconfident due to limited review variability. The kingmaker check revealed that 9 of 30 judges could single‑handedly change top‑5 placement, highlighting the need for judge‑level uncertainty modeling.

工作流

• Collect raw score sheets from organizers
• Detect and exclude flat‑liner judges
• Fit additive model iteratively for quality and leniency with ridge regularisation
• Resample each project's reviews 300 times, refit model each time, and count top‑5 occurrences
• Compute chance of top‑5 finish for each project
• Flag close calls (20–80% chance)
• Suggest additional judges for close‑call projects
• Run kingmaker check by refitting model excluding each judge in turn and flagging changes in top‑5 or first place

证据

The flat‑lining judge has a spread of zero. So does any judge who only reviewed one project. Divide by zero, and those projects can't be ranked at all.

来源

It never divides by anyone's spread, so the flat‑liner can't break it.

来源

The resampling only shuffles the reviews a project already has. With two or three reviews there just aren't many ways to shuffle, so the spread looks small.

来源

9 of 30 judges each move a top‑5 place on their own. And two of them flip first place, the one place the resampling called fairly safe.

来源