I Found Two Bugs in a Hackathon's Judging Tool. Neither Explained Why I Lost

说明与用途

问题在于评测工具中的隐藏错误导致准确率指标未被计算，导致该部分得分为零。修复 adtc-profiler 和 lm-eval-harness 中的错误后，恢复了指标并显著提升了总分。

问题

The judging tool’s accuracy calculation was silently failing, causing a zero score for the accuracy component and a lower overall score, despite the model performing well on other metrics.

结果

Fixing two silent bugs in the judging pipeline restored the accuracy metric, dramatically improving the overall score and revealing that the model’s performance was competitive with fine‑tuned peers.

工作流

• Clone the adtc-profiler and lm-eval-harness repositories
• Inspect accuracy.py for base_url usage
• Patch base_url to use http://host:port/v1/completions
• Ensure lm-eval-harness GGUF backend requests echo=true
• Run local llama-server with the model
• Execute adtc-profiler to generate submission.json
• Verify accuracy array is populated
• Recalculate total score using the hackathon formula

证据

adtc_profiler's own accuracy.py calls lm_eval with base_url=local. Not a URL.

来源

lm-eval-harness's GGUF backend expects echo=true to return logprobs for the whole echoed prompt

来源