The Incident That Never Paged: A Schema-Drift Postmortem

说明与用途

一次隐藏回归事件，模型返回文本而非JSON，默认值掩盖了错误。解决方案包括提示版本化、严格模式验证和金丝雀测试。

问题

A silent regression caused invoice totals to be wrong because a model‑backed extraction step returned prose instead of JSON, and downstream defaults masked the error.

结果

The incident was contained by reverting the prompt, but the root cause was silent defaults and lack of model‑driven tests. The fix introduced prompt versioning, strict schema validation, and golden‑file tests, eliminating the silent regression and reducing detection gap.

工作流

• 1. Detect ticket spike and reproduce locally. 2. Replay last 500 documents to quantify failure rate. 3. Revert prompt to previous version and confirm rate drops. 4. Backfill affected records with pinned prompt and compare totals. 5. Version prompts by hashing content and logging hash with each extraction. 6. Refactor parser to use strict Pydantic models and arithmetic checks. 7. Create golden file tests that call the model and assert expected structure. 8. Gate merges on passing golden file tests.

证据

First, someone edited the extraction prompt in place to make output "cleaner", with no version pin and no changelog entry, so the diff never appeared in a code review.

来源

The downstream parser treated missing fields as optional and substituted defaults instead of failing loudly, which converted a hard error into a soft data corruption.

来源

The team had unit tests for the parser, but every fixture was hand‑written valid JSON, so the tests proved the parser worked on the input it was designed for.

来源

Reverting the prompt is a repair, not a fix. The durable version has three parts that reinforce each other: version every prompt and model reference as data rather than as a string in a function, make the parser strict so malformed output fails loudly in staging, and gate merges on a golden set that actually calls a model.

来源