Nothing Reported It

说明与用途

针对小型语言模型的测试 harness 由于指标、数据加载和错误处理中的隐藏缺陷，导致结果误导。修复五个关键问题后，指标准确反映了模型性能。

问题

A test harness for evaluating small language models produced misleading results because of hidden bugs in metrics, data loading, and error handling.

结果

The corrected harness revealed that the models performed poorly on grounded QA and that earlier high scores were artifacts of flawed metrics and missing data.

工作流

• 1. Run experiments on Raspberry Pi 4 with two Ollama models across three workflows.
• 2. Observe misleading metrics: safe escalation scored high, keyword checks returned false positives, and speedup appeared due to answer length.
• 3. Identify hidden bugs: missing reference corpus, vacuous fixtures, blind spot in decay check, failure handling, and silent missing inputs.
• 4. Fix each bug: adjust metrics to penalize over‑cautious behavior, validate fixtures, account for answer length in speed measurement, classify failures correctly, and add checks for missing data.
• 5. Re‑run experiments and verify that metrics now reflect true model capabilities.

证据

The number that gave it away was sitting in a table I had already read three times.

来源

The 360M model answered escalate to 43 of 50 messages.

来源

The keyword check scored 1 of 11 for both models.

来源

Partway through the first extraction run, Ollama returned a 500 and the runner died at case seven of thirty.

来源

The function was four lines: def corpus_text(): ...

来源