Three Dead Ends in an AI Pairing Session, and the Replayable Claim Ledger That Ended It
Description and use
During an AI pairing session, engineers fixed a double‑charge bug but the assistant’s explanatory claims lacked evidence. They introduced a claim ledger, recording each claim with a command that verifies it, enabling the senior engineer to accept the patch.
Problem
A webhook handler double‑charged customers due to a retry bug, and a coding assistant’s patch was rejected because its explanatory claims were not verifiable.
Outcome
The claim ledger turned prose assertions into executable tests, enabling the senior engineer to verify all claims and accept the patch.
Workflow
- 1. Reproduce the double‑charge bug by sending a signed webhook twice.
- 2. Use the AI assistant to propose a patch and an explanation with four claims.
- 3. Attempt to review the explanation by reading the transcript, re‑rolling the prompt, and relying on a passing test suite—each approach fails to provide evidence for all claims.
- 4. Create a JSONL claim ledger, recording each claim with a command that will fail if the claim is false.
- 5. Run the ledger runner against a pinned revision to replay each claim and record verdicts.
Evidence
The bug was small: a webhook handler that double‑charged on retried deliveries.
The assistant proposed an idempotency key, a lookup before insert, and a small migration. Its explanation contained four assertions:…
The decision that survived: a claim ledger… Before the next patch, the pair wrote every assertion the assistant made into a JSONL file…
- Record date: 2026-10-08T11:10:51.510Z