I Found Two Bugs in a Hackathon's Judging Tool. Neither Explained Why I Lost

Описание и применение

Автор обнаружил две ошибки в системе оценки хакатона, исправил их, восстановил точный показатель точности и значительно повысил общий балл, но оригинальность модели осталась низкой, что привело к отказу.

Задача

The judging tool’s accuracy calculation silently failed, causing a zero score for the accuracy component and a low originality score, leading to elimination.

Результат

The author fixed two tooling bugs, recovered a valid accuracy score, and improved the total score significantly, but the originality gate still caused elimination. The case highlights the importance of understanding all scoring components, including hidden originality checks.

Рабочий процесс

• 1. Clone all semifinalist repos and review technical reports.
• 2. Inspect submission.json; notice empty accuracy array.
• 3. Identify bugs in adtc_profiler accuracy.py (incorrect base_url) and lm-eval-harness GGUF backend (echo/logprobs mismatch).
• 4. Patch both locally: use proper http://host:port URL and generate exact answer text to read confidence scores.
• 5. Run hackathon benchmark to obtain real accuracy score.
• 6. Update submission.json with calculated accuracy.
• 7. Recalculate total score.
• 8. Submit updated results.
• 9. Receive rejection due to low originality score.
• 10. Reflect on scoring formula and originality gate.

Доказательства

accuracy: []

Источник

Two bugs, both confirmed by reading the actual source

Источник

arc_easy acc_norm = 0.74

Источник

Originality Score (0–10): 3

Источник