I Found Two Bugs in a Hackathon's Judging Tool. Neither Explained Why I Lost

Description and use

The author found two bugs in the hackathon judging tool, fixed them to raise the technical score, but the originality gate remained a blocker, leading to rejection.

Problem

The author’s submission was rejected because the scoring formula penalized missing accuracy data and originality, despite the model being a stock RAG architecture that performed well on technical benchmarks.

Outcome

Fixing the evaluation bugs raised the technical score dramatically, but the originality gate remained a blocker. The case illustrates that technical excellence alone does not guarantee advancement when a hidden originality requirement exists.

Workflow

• 1. Built StacksNG with a stock Qwen2.5‑Coder‑7B model and a 780‑chunk RAG corpus.
• 2. Submitted the model with an empty accuracy array in submission.json.
• 3. Received rejection email citing low originality score and missing accuracy data.
• 4. Investigated the evaluation pipeline and discovered two bugs: (a) adtc_profiler used an invalid base_url for lm_eval; (b) lm_eval_harness expected echo=true for logprobs but the local llama-server did not provide them.
• 5. Patched both bugs locally and forced the model to generate the exact answer text to capture confidence scores.
• 6. Ran the hackathon’s benchmark and obtained arc_easy acc_norm = 0.74.
• 7. Updated the total score calculation, improving it from 10.23 to 47.23.
• 8. Realized that the originality gate was the true barrier, not the technical score.
• 9. Concluded that future submissions must modify the model (e.g., LoRA) to satisfy the originality requirement.

Evidence

submission.json:

"accuracy": []

Source

Two bugs, both confirmed by reading the actual source:

1. adtc_profiler's own accuracy.py calls lm_eval with base_url=local. Not a URL. lm-eval's GGUF backend needs http://host:port and POSTs to {base_url}/v1/completions.

Source

1. lm-eval-harness's GGUF backend expects echo=true to return logprobs for the whole echoed prompt, the old OpenAI completions behavior.

Source

Got a real number:

arc_easy acc_norm = 0.74

Source