Green Tests Can Still Be Dead Tests: Auditing Agent-Generated Suites

Описание и применение

Автоматизированный аудит тестов, созданных ИИ, с помощью мутационного тестирования выявляет пробелы в покрытии и помогает улучшить запросы к агенту.

Задача

Agent-generated test suites often appear green but may miss edge cases, leading to false confidence.

Результат

Surviving mutants pinpoint where agent tests fail, enabling prompt tuning and higher test quality.

Рабочий процесс

• 1. Ask the agent to write a test suite for a specific function.
• 2. Save the generated test file.
• 3. Run the provided mutant_audit.sh script with repo URL, source file, test file, and mutator name.
• 4. The script clones the repo, installs dependencies, copies the test file, runs the mutation tool, and outputs surviving mutants.
• 5. Review the surviving mutants to understand uncovered behaviors and adjust the prompt if needed.

Доказательства

Mutation testing offers a rigorous way to measure the difference, and a free server with a generous token allowance makes the measurement cheap enough to run on every agent-generated suite.

Источник

The audit follows a fixed sequence. The agent writes a test suite for a specific function, and the generated file is saved for inspection.

Источник