I Found Two Bugs in a Hackathon's Judging Tool. Neither Explained Why I Lost

Описание и применение

Автор обнаружил две ошибки в системе оценки хакатона, исправил их и получил реальный показатель точности, но оригинальность модели осталась низкой, что привело к отказу.

Задача

The author’s submission was rejected because the scoring formula penalized missing accuracy data and originality, despite the model being a stock RAG architecture that performed competitively.

Результат

The author’s debugging effort raised the accuracy score dramatically, but the originality gate remained a blocker. The case illustrates the importance of understanding all scoring components, including hidden gates.

Рабочий процесс

• Built StacksNG with Qwen2.5‑Coder‑7B and a 780‑chunk RAG corpus.
• Submitted the model with an empty accuracy array.
• Received rejection email citing low originality and missing accuracy.
• Cloned all semifinalist repos and noted that all used fine‑tuned models.
• Identified two bugs in adtc_profiler and lm-evaluation-harness that prevented accuracy calculation.
• Patched the bugs locally and reran the benchmark to obtain a real accuracy score.
• Re‑submitted the updated accuracy data.
• Still failed due to originality gate.
• Analyzed scoring formula and discovered the originality gate was not reflected in the rubric.
• Concluded that a model modification (e.g., LoRA) would be required to pass the originality gate.

Доказательства

My own submission.json:

"accuracy": []

Источник

I patched both locally. Instead of trusting the broken echo behavior, I forced the model to generate the exact answer text (a grammar rule constraining what it's allowed to output) and read its confidence scores off that generation directly.

Источник

Originality Score (0–10): **3**

Источник