Your AI Wrote Tests With 100% Coverage. They Catch 4% of Real Bugs

Описание и применение

AI‑генерируемые тесты могут достигать 100 % покрытия, но пропускать большинство реальных ошибок. Независимая проверка или мутационное тестирование выявляют недостающие сценарии, повышая надёжность тестов.

Задача

AI-generated tests can achieve 100% line coverage yet miss most real bugs, as coverage alone does not guarantee test depth.

Результат

Using independent verification or mutation testing exposes the gap between coverage and defect detection, leading to more reliable AI-generated tests.

Рабочий процесс

• Generate test suite with AI
• Run AI-generated tests and observe 100% coverage
• Apply mutation testing or independent verification suite
• Identify surviving mutants or failed verification tests
• Add or adjust tests to cover missed scenarios

Доказательства

100% line coverage, 4% mutation score. Every line executed. 96% of seeded bugs slipped through anyway.

Источник

The checker produced 554 test cases. The resulting sweep found 38 defects — including a severity-one bug in session handling that the builder's own tests had missed.

Источник

You don't need a full mutation testing setup to feel this gap in under ten minutes. I built a free, runnable lab that does the same thing by hand: four bugs seeded into working-looking code.

Источник