358 pull requests that changed tests: agents rarely weakened them. They bent the code instead.

Description and use

Analysis of 2026 PRs shows agents do not weaken tests more than humans; deliberate check changes are common and detectable with RepoPilot.

Problem

Determine whether coding agents weaken tests by skipping or deleting them, and build a guard to detect such behavior.

Outcome

No meaningful difference between agents and humans in unexplained test weakening; deliberate changes to checks are common and detectable by RepoPilot.

Workflow

• Collect PRs and label them manually
• Analyze PRs for test modifications
• Build RepoPilot review tool to list changes to checks
• Run RepoPilot review on PRs
• Use findings to inform guard development

Evidence

198 merged, approved PRs (84 from coding agents, 114 from humans); 98 more merged PRs from an earlier window, held out until the tool was done; 62 agent PRs that reviewers closed without merging.

Source

Unexplained weakening is rare for everyone. A skipped test, a deleted assertion, a new suppression, or a relaxed CI step that nothing in the PR explains: 0 of 84 and 0 of 42 merged agent PRs, 0 of 114 and 2 of 56 human PRs, 1 of 62 closed agent PRs.

Source

Deliberate changes to checks are common. 19-39% of PRs changed or removed a check on purpose, usually because the behavior under test changed.

Source

RepoPilot review lists every place a change touched the checks that judge it: focused or skipped tests, removed tests, tests that lost assertions, new lint/type/coverage suppressions, relaxed CI or tool gates, and new entries in RepoPilot's own suppression file.

Source