I built a hackathon judging app and it was too confident

Description and use

The hackathon judging app used an additive model and resampling but was overconfident due to limited review variability. Judges who could single‑handedly change the top‑5 were identified.

Problem

The app produced overly confident ranking results because the resampling method only shuffled existing reviews and did not account for judge variability, leading to inaccurate uncertainty estimates.

Outcome

The additive model outperformed plain average and z‑score methods (Spearman 0.813 vs 0.697 and 0.778). Resampling provided useful uncertainty signals, though it was overconfident due to limited review variability. The kingmaker check revealed that 9 of 30 judges could single‑handedly change top‑5 placement, highlighting the need for judge‑level uncertainty modeling.

Workflow

• Collect raw score sheets from organizers
• Detect and exclude flat‑liner judges
• Fit additive model iteratively for quality and leniency with ridge regularisation
• Resample each project's reviews 300 times, refit model each time, and count top‑5 occurrences
• Compute chance of top‑5 finish for each project
• Flag close calls (20–80% chance)
• Suggest additional judges for close‑call projects
• Run kingmaker check by refitting model excluding each judge in turn and flagging changes in top‑5 or first place

Evidence

The flat‑lining judge has a spread of zero. So does any judge who only reviewed one project. Divide by zero, and those projects can't be ranked at all.

Source

It never divides by anyone's spread, so the flat‑liner can't break it.

Source

The resampling only shuffles the reviews a project already has. With two or three reviews there just aren't many ways to shuffle, so the spread looks small.

Source

9 of 30 judges each move a top‑5 place on their own. And two of them flip first place, the one place the resampling called fairly safe.

Source