I Found Two Bugs in a Hackathon's Judging Tool. Neither Explained Why I Lost

说明与用途

作者在黑客松评估工具中发现了两个错误，修复后获得了真实的准确率分数，显著提升了总分，但原创性门槛仍未通过。

问题

The author’s submission was rejected because of missing accuracy data and an originality gate, despite having a functional RAG system and correct model behavior.

结果

Fixing two tooling bugs allowed the author to compute a valid accuracy score, boosting the overall score dramatically, but the originality gate still prevented advancement.

工作流

• 1. Built StacksNG with Qwen2.5-Coder-7B and a 780‑chunk RAG corpus.
• 2. Submitted with empty accuracy array in submission.json.
• 3. Received rejection email citing low originality score.
• 4. Cloned all semifinalist repos and reviewed technical reports.
• 5. Identified two bugs in adtc_profiler and lm-eval-harness.
• 6. Patched the bugs locally and reran the official benchmark to obtain a real accuracy score.
• 7. Updated submission.json with the new accuracy value.
• 8. Re‑submitted and noted the score increase, but originality gate remained.
• 9. Filed upstream issues for the bugs (adtc-profiler#4, lm-evaluation-harness#4158).

证据

Two bugs, both confirmed by reading the actual source: 1) adtc_profiler’s own accuracy.py calls lm_eval with base_url=local. 2) lm-eval-harness’s GGUF backend expects echo=true to return logprobs for the whole echoed prompt.

来源

Got a real number: arc_easy acc_norm = 0.74

来源