I Found Two Bugs in a Hackathon's Judging Tool. Neither Explained Why I Lost

Описание и применение

Автор обнаружил две ошибки в системе оценки хакатона, исправил их и получил реальный показатель точности, что значительно повысило общий балл, но оригинальность модели всё равно не прошла проверку.

Задача

The author’s submission was rejected because of missing accuracy data and an originality gate, despite having a functional RAG system and correct model behavior.

Результат

Fixing two tooling bugs allowed the author to compute a valid accuracy score, boosting the overall score dramatically, but the originality gate still prevented advancement.

Рабочий процесс

• 1. Built StacksNG with Qwen2.5-Coder-7B and a 780‑chunk RAG corpus.
• 2. Submitted with empty accuracy array in submission.json.
• 3. Received rejection email citing low originality score.
• 4. Cloned all semifinalist repos and reviewed technical reports.
• 5. Identified two bugs in adtc_profiler and lm-eval-harness.
• 6. Patched the bugs locally and reran the official benchmark to obtain a real accuracy score.
• 7. Updated submission.json with the new accuracy value.
• 8. Re‑submitted and noted the score increase, but originality gate remained.
• 9. Filed upstream issues for the bugs (adtc-profiler#4, lm-evaluation-harness#4158).

Доказательства

Two bugs, both confirmed by reading the actual source: 1) adtc_profiler’s own accuracy.py calls lm_eval with base_url=local. 2) lm-eval-harness’s GGUF backend expects echo=true to return logprobs for the whole echoed prompt.

Источник

Got a real number: arc_easy acc_norm = 0.74

Источник