Green Tests Can Still Be Dead Tests: Auditing Agent-Generated Suites

说明与用途

使用变异测试对 AI 生成的测试进行自动化审核，发现覆盖缺口并指导提示改进。

问题

Agent-generated test suites often appear green but may miss edge cases, leading to false confidence.

结果

Surviving mutants pinpoint where agent tests fail, enabling prompt tuning and higher test quality.

工作流

• 1. Ask the agent to write a test suite for a specific function.
• 2. Save the generated test file.
• 3. Run the provided mutant_audit.sh script with repo URL, source file, test file, and mutator name.
• 4. The script clones the repo, installs dependencies, copies the test file, runs the mutation tool, and outputs surviving mutants.
• 5. Review the surviving mutants to understand uncovered behaviors and adjust the prompt if needed.

证据

Mutation testing offers a rigorous way to measure the difference, and a free server with a generous token allowance makes the measurement cheap enough to run on every agent-generated suite.

来源

The audit follows a fixed sequence. The agent writes a test suite for a specific function, and the generated file is saved for inspection.

来源