The Test That Can't Fail: Reading the Test Diff of an Agent-Generated PR

Description and use

Inspecting the test diff in AI-generated PRs reveals tests that do not verify new behavior, using diff splitting and mutation testing.

Problem

Agent-generated PRs often contain tests that do not actually verify new behavior, leading to false confidence in CI results.

Outcome

By focusing on the test diff and applying mutation testing, reviewers can detect tests that do not actually verify new behavior, reducing the risk of false positives in CI.

Workflow

• 1. Split the diff by kind, separating production files from test files.
• 2. Write a one-sentence description of the new production behavior.
• 3. Locate the assertion that would fail if the behavior were absent.
• 4. Break the production line in a scratch copy and rerun only the new tests.
• 5. Revert any tests that do not fail when the implementation is broken.
• 6. Run a mutation smoke test that alters numeric literals and booleans in production files and reruns the new tests to find survivors.

Evidence

Agent-generated PRs fail review in a specific way. The production code looks reasonable, the CI is green, and the new tests prove nothing — because they mock the one function the PR changed.

Source

Strip the tests out and the diff is usually small. That is the tell.

Source

The failure mode is double counting. The agent asserts X in the PR body, asserts X in a test name, and you count three pieces of evidence where there is one.

Source

This script has not been executed by the article author. Treat the paths as placeholders and run it in a throwaway worktree, never in your working tree.

Source