The Chat Rendered a Perfect Diff. sha256sum Did Not Budge

Description and use

Hash verification and explicit test discovery prevented a 48‑hour debugging loop caused by a model’s misleading diff output.

Problem

Relying on a model’s rendered unified diff can mislead developers into thinking code changes were applied when they were not, causing test failures and wasted debugging time.

Outcome

Adding a hash‑based gate and explicit test collection checks prevented a 48‑hour debugging loop, ensuring that code changes were truly applied before tests ran.

Workflow

• 1. Run pytest locally on the updated clone and observe green output.
• 2. On the remote server, run git status and sha256sum src/billing.py to confirm the file is unchanged.
• 3. Run pytest --collect-only -q to ensure tests are discovered and node IDs are present.
• 4. Execute a snapshot script (snapshot.sh) to hash all relevant files before and after the model’s edit.
• 5. Compare the before and after hash files; if unchanged, abort the run.
• 6. If hashes differ and tests are collected, proceed with pytest execution.
• 7. If tests pass, accept the changes; otherwise, investigate further.

Evidence

I pasted a failing assertion, asked for a fix, and received a confident unified diff against src/billing.py.

Source

When diff printed nothing, I finally understood the weekend in one glance.

Source