Nothing Reported It

Описание и применение

Тестовый harness для небольших языковых моделей выдавал неверные результаты из‑за скрытых ошибок в метриках, загрузке данных и обработке ошибок. После исправления пяти ключевых проблем показатели стали достоверными.

Задача

A test harness for evaluating small language models produced misleading results because of hidden bugs in metrics, data loading, and error handling.

Результат

The corrected harness revealed that the models performed poorly on grounded QA and that earlier high scores were artifacts of flawed metrics and missing data.

Рабочий процесс

• 1. Run experiments on Raspberry Pi 4 with two Ollama models across three workflows.
• 2. Observe misleading metrics: safe escalation scored high, keyword checks returned false positives, and speedup appeared due to answer length.
• 3. Identify hidden bugs: missing reference corpus, vacuous fixtures, blind spot in decay check, failure handling, and silent missing inputs.
• 4. Fix each bug: adjust metrics to penalize over‑cautious behavior, validate fixtures, account for answer length in speed measurement, classify failures correctly, and add checks for missing data.
• 5. Re‑run experiments and verify that metrics now reflect true model capabilities.

Доказательства

The number that gave it away was sitting in a table I had already read three times.

Источник

The 360M model answered escalate to 43 of 50 messages.

Источник

The keyword check scored 1 of 11 for both models.

Источник

Partway through the first extraction run, Ollama returned a 500 and the runner died at case seven of thirty.

Источник

The function was four lines: def corpus_text(): ...

Источник