Reachability Receipts: Prove an Agent Patch's Green Build Came From Tests That Touched It

Description and use

The receipt format records which tests cover the changed lines, preventing flaky tests from being used as evidence for a green build.

Problem

A green build can be reported even when no test actually exercised the patched lines, allowing flaky tests to be frozen and falsely used as evidence.

Outcome

The receipt format explicitly records which tests touched the patched lines, preventing flaky tests from being used as evidence for a green build.

Workflow

• Derive changed line ranges with git diff -U0 and count hunks
• Collect per-test line hits using coverage.py with dynamic_context=test_function and --show-contexts
• Convert coverage JSON to JSONL format for the receipt
• Run the receipt script to intersect changed lines with test hits and check against the freeze set

Evidence

A green build is a fact about your test suite, not about the patch.

Source

Everything else is bookkeeping. The verdict comes from intersecting the first two, then asking the third whether that intersection is trustworthy.

Source