Outside the Assertion: The Retry Budget a Pairing Hour Refused to Inline

Description and use

During a pairing session, a test for GET /orders/:id intermittently passed after a rerun, raising concerns about flakiness. The decision was made to run the test once and handle retries as separate commands with a written budget and receipt of seed, command, exit code, and output hash.

Problem

A test for a service endpoint intermittently passes after a rerun, raising uncertainty about flakiness and the reliability of the test harness.

Outcome

The pairing session concluded with a documented rule that separates test execution from retries, ensuring reproducibility and traceability of failures.

Workflow

• 1. Run the test once and record the result.
• 2. If a retry is needed, execute a separate command with a defined retry budget.
• 3. Record seed, command, exit code, and hash of output for each attempt.
• 4. Ensure a second process reproduces the failure before code changes.

Evidence

The contract check failed once, then passed on a local rerun.

Source

The senior asked three questions before anyone edited the helper.

Source

The test process runs the check once.

Source