I Found Two Bugs in a Hackathon's Judging Tool. Neither Explained Why I Lost

Description and use

The author found two bugs in the hackathon judging tool, fixed them, calculated a real accuracy score, which boosted the overall score, but the originality gate still failed.

Problem

The author’s submission was rejected because of missing accuracy data and an originality gate, despite having a functional RAG system and correct model behavior.

Outcome

Fixing two tooling bugs allowed the author to compute a valid accuracy score, boosting the overall score dramatically, but the originality gate still prevented advancement.

Workflow

• 1. Built StacksNG with Qwen2.5-Coder-7B and a 780‑chunk RAG corpus.
• 2. Submitted with empty accuracy array in submission.json.
• 3. Received rejection email citing low originality score.
• 4. Cloned all semifinalist repos and reviewed technical reports.
• 5. Identified two bugs in adtc_profiler and lm-eval-harness.
• 6. Patched the bugs locally and reran the official benchmark to obtain a real accuracy score.
• 7. Updated submission.json with the new accuracy value.
• 8. Re‑submitted and noted the score increase, but originality gate remained.
• 9. Filed upstream issues for the bugs (adtc-profiler#4, lm-evaluation-harness#4158).

Evidence

Two bugs, both confirmed by reading the actual source: 1) adtc_profiler’s own accuracy.py calls lm_eval with base_url=local. 2) lm-eval-harness’s GGUF backend expects echo=true to return logprobs for the whole echoed prompt.

Source

Got a real number: arc_easy acc_norm = 0.74

Source