I Found Two Bugs in a Hackathon's Judging Tool. Neither Explained Why I Lost

说明与用途

作者在黑客松评估工具中发现并修复了两处错误，提升了技术评分，但原创性门槛仍未通过，导致被拒。

问题

The author’s submission was rejected because the scoring formula penalized missing accuracy data and originality, despite the model being a stock RAG architecture that performed well on technical benchmarks.

结果

Fixing the evaluation bugs raised the technical score dramatically, but the originality gate remained a blocker. The case illustrates that technical excellence alone does not guarantee advancement when a hidden originality requirement exists.

工作流

• 1. Built StacksNG with a stock Qwen2.5‑Coder‑7B model and a 780‑chunk RAG corpus.
• 2. Submitted the model with an empty accuracy array in submission.json.
• 3. Received rejection email citing low originality score and missing accuracy data.
• 4. Investigated the evaluation pipeline and discovered two bugs: (a) adtc_profiler used an invalid base_url for lm_eval; (b) lm_eval_harness expected echo=true for logprobs but the local llama-server did not provide them.
• 5. Patched both bugs locally and forced the model to generate the exact answer text to capture confidence scores.
• 6. Ran the hackathon’s benchmark and obtained arc_easy acc_norm = 0.74.
• 7. Updated the total score calculation, improving it from 10.23 to 47.23.
• 8. Realized that the originality gate was the true barrier, not the technical score.
• 9. Concluded that future submissions must modify the model (e.g., LoRA) to satisfy the originality requirement.

证据

submission.json:

"accuracy": []

来源

Two bugs, both confirmed by reading the actual source:

1. adtc_profiler's own accuracy.py calls lm_eval with base_url=local. Not a URL. lm-eval's GGUF backend needs http://host:port and POSTs to {base_url}/v1/completions.

来源

1. lm-eval-harness's GGUF backend expects echo=true to return logprobs for the whole echoed prompt, the old OpenAI completions behavior.

来源

Got a real number:

arc_easy acc_norm = 0.74

来源