I Found Two Bugs in a Hackathon's Judging Tool. Neither Explained Why I Lost

Description and use

The issue was silent bugs in the judging tools that caused the accuracy metric to be omitted, resulting in a zero score for that component. Fixing the bugs in adtc-profiler and lm-eval-harness restored the metric and boosted the overall score.

Problem

The judging tool’s accuracy calculation was silently failing, causing a zero score for the accuracy component and a lower overall score, despite the model performing well on other metrics.

Outcome

Fixing two silent bugs in the judging pipeline restored the accuracy metric, dramatically improving the overall score and revealing that the model’s performance was competitive with fine‑tuned peers.

Workflow

• Clone the adtc-profiler and lm-eval-harness repositories
• Inspect accuracy.py for base_url usage
• Patch base_url to use http://host:port/v1/completions
• Ensure lm-eval-harness GGUF backend requests echo=true
• Run local llama-server with the model
• Execute adtc-profiler to generate submission.json
• Verify accuracy array is populated
• Recalculate total score using the hackathon formula

Evidence

adtc_profiler's own accuracy.py calls lm_eval with base_url=local. Not a URL.

Source

lm-eval-harness's GGUF backend expects echo=true to return logprobs for the whole echoed prompt

Source