CI only ever runs the new tests against the new code

说明与用途

CI 仅在新代码上运行新测试，若测试随代码一起修改，易隐藏错误。pr‑witness 通过在新代码上运行旧测试来发现隐藏缺陷。

问题

When a pull request edits both code and its tests, CI only runs the new tests against the new code, allowing a developer or agent to hide bugs by synchronously changing tests to match the buggy code.

结果

pr‑witness uncovered a hidden bug that CI missed, demonstrating that running old tests against new code is essential when tests can be edited alongside code.

工作流

• 1. Pull request contains code and test changes.
• 2. CI runs the pull request’s tests against the pull request’s code (new tests vs new code).
• 3. Build passes because tests were altered to match the buggy code.
• 4. pr‑witness executes the base branch’s tests against the head branch’s code (old tests vs new code).
• 5. pr‑witness reports any discrepancies, such as a test that now fails.
• 6. The report is signed with GitHub Artifact Attestations for integrity.

证据

CI runs the pull request's tests against the pull request's code. Both sides of that comparison were written by the same author, in the same commit. If the author — human or agent — edits both halves consistently, the result is green by construction.

来源

The combination nobody runs is the old tests against the new code.

来源

$ pr-witness --base main base head crossrun passed 3 3 2 failed 0 0 1 Claims ! "3 tests pass" — true, but 1 test that passed on the base branch fails against this code ✗ "no breaking changes" — 1 base-branch test now fails !! [crossrun] tests/test_dates::test_monday

来源