Pin the Crash in a Characterization Test Before You Patch Upstream

Description and use

The workflow shows how to capture a crash in a test before modifying production code, enabling maintainers to replay and verify the fix.

Problem

First‑time contributors often submit patches that only show a successful path, leaving the original crash undocumented and hard for maintainers to reproduce.

Outcome

The workflow ensures that the original crash is captured as an executable oracle, making it easy for maintainers to reproduce and verify the fix, and preventing regressions.

Workflow

• Clone the repository and reproduce the crash locally.
• Write a test that records the current crash behavior (e.g., non‑zero exit, exception message).
• Commit the test to a branch that still matches upstream main without modifying production code.
• Create a helper script to run the test on main and on the patched commit, showing the pass/fail flip.
• Add labels (oracle-before / oracle-after) in the PR body to document the change.
• Include a CSV grid comparing crash status across release tags, main, and the patched commit.
• Submit the PR with the test and helper scripts attached, ensuring reviewers can replay the failure.

Evidence

The pull request still stalled because maintainers could not replay the old failure against the new tree.

Source

A useful oracle fails on the unpatched tree and passes on the patched commit for one named input.

Source