The Spike Dies When Two Runs Disagree

Описание и применение

В статье описывается автоматизированный рецепт проведения двух‑поточного спайка с ограничением времени, позволяющий получить воспроизводимое доказательство гипотезы о причине сбоя теста, прежде чем применять исправление от ИИ.

Задача

A worker that retries a failed job three times sometimes performs the same side effect twice, causing uncertainty about the cause of the failure and distrust in AI-assisted patches.

Результат

The spike harness produces a single verdict that either confirms the hypothesis or rejects it, providing reproducible evidence before any patch is applied.

Рабочий процесс

• Write a single‑sentence hypothesis, prediction, and kill criteria.
• Create a bash harness that sets a 90‑minute deadline shared by both runs.
• For each run: create a detached worktree at HEAD, record the base revision, run the failing test to confirm baseline failure.
• Run the AI assistant once with the unchanged prompt, capture the output patch.
• Apply the patch, run the test again, record pass/fail and diff stat.
• After both runs, run a Python verdict script to produce a single verdict file.
• Clean up worktrees with git worktree remove --force and git worktree prune.

Доказательства

The spike begins with a single sentence that predicts a specific, falsifiable outcome.

Источник

Two runs do not prove correctness, and the protocol never claims they do.

Источник

The script below creates the worktrees, records the base revision, checks that the target test actually fails before the assistant touches anything, and saves each attempt's patch, diff stat, and test result.

Источник

Two details are easy to get wrong. timeout measures wall clock, so a run that hangs on a prompt cannot borrow time from the next one; the semantics are documented in the coreutils manual.

Источник