The Incident That Never Paged: A Schema-Drift Postmortem

Description and use

A silent regression where a model returned prose instead of JSON, masked by default values. The fix added prompt versioning, strict schema validation, and golden tests.

Problem

A silent regression caused invoice totals to be wrong because a model‑backed extraction step returned prose instead of JSON, and downstream defaults masked the error.

Outcome

The incident was contained by reverting the prompt, but the root cause was silent defaults and lack of model‑driven tests. The fix introduced prompt versioning, strict schema validation, and golden‑file tests, eliminating the silent regression and reducing detection gap.

Workflow

• 1. Detect ticket spike and reproduce locally. 2. Replay last 500 documents to quantify failure rate. 3. Revert prompt to previous version and confirm rate drops. 4. Backfill affected records with pinned prompt and compare totals. 5. Version prompts by hashing content and logging hash with each extraction. 6. Refactor parser to use strict Pydantic models and arithmetic checks. 7. Create golden file tests that call the model and assert expected structure. 8. Gate merges on passing golden file tests.

Evidence

First, someone edited the extraction prompt in place to make output "cleaner", with no version pin and no changelog entry, so the diff never appeared in a code review.

Source

The downstream parser treated missing fields as optional and substituted defaults instead of failing loudly, which converted a hard error into a soft data corruption.

Source

The team had unit tests for the parser, but every fixture was hand‑written valid JSON, so the tests proved the parser worked on the input it was designed for.

Source

Reverting the prompt is a repair, not a fix. The durable version has three parts that reinforce each other: version every prompt and model reference as data rather than as a string in a function, make the parser strict so malformed output fails loudly in staging, and gate merges on a golden set that actually calls a model.

Source