CI only ever runs the new tests against the new code

Описание и применение

CI запускает новые тесты только на новый код, позволяя скрыть ошибки, если тесты меняются вместе с кодом. pr‑witness решает эту проблему, выполняя старые тесты на новый код, выявляя скрытые баги.

Задача

When a pull request edits both code and its tests, CI only runs the new tests against the new code, allowing a developer or agent to hide bugs by synchronously changing tests to match the buggy code.

Результат

pr‑witness uncovered a hidden bug that CI missed, demonstrating that running old tests against new code is essential when tests can be edited alongside code.

Рабочий процесс

• 1. Pull request contains code and test changes.
• 2. CI runs the pull request’s tests against the pull request’s code (new tests vs new code).
• 3. Build passes because tests were altered to match the buggy code.
• 4. pr‑witness executes the base branch’s tests against the head branch’s code (old tests vs new code).
• 5. pr‑witness reports any discrepancies, such as a test that now fails.
• 6. The report is signed with GitHub Artifact Attestations for integrity.

Доказательства

CI runs the pull request's tests against the pull request's code. Both sides of that comparison were written by the same author, in the same commit. If the author — human or agent — edits both halves consistently, the result is green by construction.

Источник

The combination nobody runs is the old tests against the new code.

Источник

$ pr-witness --base main base head crossrun passed 3 3 2 failed 0 0 1 Claims ! "3 tests pass" — true, but 1 test that passed on the base branch fails against this code ✗ "no breaking changes" — 1 base-branch test now fails !! [crossrun] tests/test_dates::test_monday

Источник