Three Dead Ends in an AI Pairing Session, and the Replayable Claim Ledger That Ended It

Описание и применение

В ходе сессии с ИИ два инженера исправили ошибку двойного списания, но объяснение исправления не было подтверждено. В итоге они создали «журнал заявлений», где каждое утверждение сопровождается командой, проверяющей его истинность, что позволило убедиться в корректности патча.

Задача

A webhook handler double‑charged customers due to a retry bug, and a coding assistant’s patch was rejected because its explanatory claims were not verifiable.

Результат

The claim ledger turned prose assertions into executable tests, enabling the senior engineer to verify all claims and accept the patch.

Рабочий процесс

• 1. Reproduce the double‑charge bug by sending a signed webhook twice.
• 2. Use the AI assistant to propose a patch and an explanation with four claims.
• 3. Attempt to review the explanation by reading the transcript, re‑rolling the prompt, and relying on a passing test suite—each approach fails to provide evidence for all claims.
• 4. Create a JSONL claim ledger, recording each claim with a command that will fail if the claim is false.
• 5. Run the ledger runner against a pinned revision to replay each claim and record verdicts.

Доказательства

The bug was small: a webhook handler that double‑charged on retried deliveries.

Источник

The assistant proposed an idempotency key, a lookup before insert, and a small migration. Its explanation contained four assertions:…

Источник

The decision that survived: a claim ledger… Before the next patch, the pair wrote every assertion the assistant made into a JSONL file…

Источник