The Spike Dies When Two Runs Disagree

说明与用途

文章介绍了一个两次运行、时间限制的自动化实验方案，在应用 AI 生成的补丁之前，提供可复现的证据来验证关于测试失败原因的假设。

问题

A worker that retries a failed job three times sometimes performs the same side effect twice, causing uncertainty about the cause of the failure and distrust in AI-assisted patches.

结果

The spike harness produces a single verdict that either confirms the hypothesis or rejects it, providing reproducible evidence before any patch is applied.

工作流

• Write a single‑sentence hypothesis, prediction, and kill criteria.
• Create a bash harness that sets a 90‑minute deadline shared by both runs.
• For each run: create a detached worktree at HEAD, record the base revision, run the failing test to confirm baseline failure.
• Run the AI assistant once with the unchanged prompt, capture the output patch.
• Apply the patch, run the test again, record pass/fail and diff stat.
• After both runs, run a Python verdict script to produce a single verdict file.
• Clean up worktrees with git worktree remove --force and git worktree prune.

证据

The spike begins with a single sentence that predicts a specific, falsifiable outcome.

来源

Two runs do not prove correctness, and the protocol never claims they do.

来源

The script below creates the worktrees, records the base revision, checks that the target test actually fails before the assistant touches anything, and saves each attempt's patch, diff stat, and test result.

来源

Two details are easy to get wrong. timeout measures wall clock, so a run that hangs on a prompt cannot borrow time from the next one; the semantics are documented in the coreutils manual.

来源