358 pull requests that changed tests: agents rarely weakened them. They bent the code instead.

Описание и применение

Изучение 2026 PR‑ов показало, что агенты не уменьшают тесты чаще, чем люди; изменения проверок встречаются часто и их можно обнаружить с помощью RepoPilot.

Задача

Determine whether coding agents weaken tests by skipping or deleting them, and build a guard to detect such behavior.

Результат

No meaningful difference between agents and humans in unexplained test weakening; deliberate changes to checks are common and detectable by RepoPilot.

Рабочий процесс

• Collect PRs and label them manually
• Analyze PRs for test modifications
• Build RepoPilot review tool to list changes to checks
• Run RepoPilot review on PRs
• Use findings to inform guard development

Доказательства

198 merged, approved PRs (84 from coding agents, 114 from humans); 98 more merged PRs from an earlier window, held out until the tool was done; 62 agent PRs that reviewers closed without merging.

Источник

Unexplained weakening is rare for everyone. A skipped test, a deleted assertion, a new suppression, or a relaxed CI step that nothing in the PR explains: 0 of 84 and 0 of 42 merged agent PRs, 0 of 114 and 2 of 56 human PRs, 1 of 62 closed agent PRs.

Источник

Deliberate changes to checks are common. 19-39% of PRs changed or removed a check on purpose, usually because the behavior under test changed.

Источник

RepoPilot review lists every place a change touched the checks that judge it: focused or skipped tests, removed tests, tests that lost assertions, new lint/type/coverage suppressions, relaxed CI or tool gates, and new entries in RepoPilot's own suppression file.

Источник