Build an OSS Patch Allowlist From One Failing Test

Description and use

Create a single test that reproduces the bug, collect coverage only for that test, and generate an allowlist of files that may change. Reject any PR that touches files outside the list.

Problem

Issue threads often lead to patches that touch unrelated code, causing future regressions. A single failing test can be used as a precise merge oracle, but relying on issue comments is unreliable.

Outcome

The workflow ensures that a patch only touches code exercised by the failing test, reducing the risk of regressions and making the PR easier to review.

Workflow

• 1. Ensure a clean working tree and fetch the target branch.
• 2. Create a single test that reproduces the bug and name it after the issue number.
• 3. Run that test to confirm it fails as expected.
• 4. Collect coverage for that test only and generate an allowlist of covered files.
• 5. Add the test file itself to the allowlist.
• 6. Run a script that compares the current diff against the allowlist and exits with an error if any extra paths are present.
• 7. Edit only the files in the allowlist, re‑running the test after each change until it passes.

Evidence

An OSS patch is ready only after one named test flips. That named failing test is the merge oracle.

Source

Coverage names every path the test actually executes. Files outside that map stay out of the commit.

Source