CI only ever runs the new tests against the new code

Description and use

CI runs new tests against new code, letting bugs slip when tests are edited with code. pr‑witness runs old tests on new code, catching hidden defects.

Problem

When a pull request edits both code and its tests, CI only runs the new tests against the new code, allowing a developer or agent to hide bugs by synchronously changing tests to match the buggy code.

Outcome

pr‑witness uncovered a hidden bug that CI missed, demonstrating that running old tests against new code is essential when tests can be edited alongside code.

Workflow

• 1. Pull request contains code and test changes.
• 2. CI runs the pull request’s tests against the pull request’s code (new tests vs new code).
• 3. Build passes because tests were altered to match the buggy code.
• 4. pr‑witness executes the base branch’s tests against the head branch’s code (old tests vs new code).
• 5. pr‑witness reports any discrepancies, such as a test that now fails.
• 6. The report is signed with GitHub Artifact Attestations for integrity.

Evidence

CI runs the pull request's tests against the pull request's code. Both sides of that comparison were written by the same author, in the same commit. If the author — human or agent — edits both halves consistently, the result is green by construction.

Source

The combination nobody runs is the old tests against the new code.

Source

$ pr-witness --base main base head crossrun passed 3 3 2 failed 0 0 1 Claims ! "3 tests pass" — true, but 1 test that passed on the base branch fails against this code ✗ "no breaking changes" — 1 base-branch test now fails !! [crossrun] tests/test_dates::test_monday

Source