The Spike Dies When Two Runs Disagree

Description and use

The article presents an automated recipe for a two‑run, time‑boxed spike that yields reproducible evidence for a hypothesis about a flaky test before applying an AI‑generated patch.

Problem

A worker that retries a failed job three times sometimes performs the same side effect twice, causing uncertainty about the cause of the failure and distrust in AI-assisted patches.

Outcome

The spike harness produces a single verdict that either confirms the hypothesis or rejects it, providing reproducible evidence before any patch is applied.

Workflow

• Write a single‑sentence hypothesis, prediction, and kill criteria.
• Create a bash harness that sets a 90‑minute deadline shared by both runs.
• For each run: create a detached worktree at HEAD, record the base revision, run the failing test to confirm baseline failure.
• Run the AI assistant once with the unchanged prompt, capture the output patch.
• Apply the patch, run the test again, record pass/fail and diff stat.
• After both runs, run a Python verdict script to produce a single verdict file.
• Clean up worktrees with git worktree remove --force and git worktree prune.

Evidence

The spike begins with a single sentence that predicts a specific, falsifiable outcome.

Source

Two runs do not prove correctness, and the protocol never claims they do.

Source

The script below creates the worktrees, records the base revision, checks that the target test actually fails before the assistant touches anything, and saves each attempt's patch, diff stat, and test result.

Source

Two details are easy to get wrong. timeout measures wall clock, so a run that hangs on a prompt cannot borrow time from the next one; the semantics are documented in the coreutils manual.

Source