The Free Model Wrote Tests That Passed. They Still Missed a Bug

Описание и применение

LLM‑генерируемые тесты могут проходить, но пропускать критические граничные случаи. Добавление покрытия выявило ошибку в функции парсинга порта, после чего исправление привело к 100% покрытию ветвей.

Задача

Model‑generated unit tests can pass while still missing critical edge cases, leading to undetected bugs.

Результат

Coverage instrumentation uncovered a critical edge case missed by LLM‑generated tests, leading to a bug fix and 100% branch coverage.

Рабочий процесс

• Generate tests with a free LLM (MonkeyCode).
• Compile and run the tests with a plain build.
• Compile and run the same tests with coverage instrumentation (gcov).
• Review the coverage report to identify untested branches.
• Add missing tests for uncovered branches.
• Re‑run tests and coverage to confirm all branches are exercised.
• Apply fixes to the code and verify full coverage.

Доказательства

The coverage build uses the same test binary. No extra test code is needed. The only cost is a recompile and a few seconds of runtime.

Источник

The model’s tests never called parse_port with an empty string.

Источник