The Free Model Wrote Tests That Passed. They Still Missed a Bug

Description and use

LLM‑generated tests can pass while missing critical edge cases. Adding coverage revealed a bug in a port‑parsing function, which was fixed, achieving 100% branch coverage.

Problem

Model‑generated unit tests can pass while still missing critical edge cases, leading to undetected bugs.

Outcome

Coverage instrumentation uncovered a critical edge case missed by LLM‑generated tests, leading to a bug fix and 100% branch coverage.

Workflow

• Generate tests with a free LLM (MonkeyCode).
• Compile and run the tests with a plain build.
• Compile and run the same tests with coverage instrumentation (gcov).
• Review the coverage report to identify untested branches.
• Add missing tests for uncovered branches.
• Re‑run tests and coverage to confirm all branches are exercised.
• Apply fixes to the code and verify full coverage.

Evidence

The coverage build uses the same test binary. No extra test code is needed. The only cost is a recompile and a few seconds of runtime.

Source

The model’s tests never called parse_port with an empty string.

Source