The Incident That Never Paged: A Schema-Drift Postmortem

Описание и применение

Случай скрытой регрессии, где модель возвращала текст вместо JSON, а дефолтные значения маскировали ошибку. Решение включало версионирование промптов, строгую валидацию схем и золотые тесты.

Задача

A silent regression caused invoice totals to be wrong because a model‑backed extraction step returned prose instead of JSON, and downstream defaults masked the error.

Результат

The incident was contained by reverting the prompt, but the root cause was silent defaults and lack of model‑driven tests. The fix introduced prompt versioning, strict schema validation, and golden‑file tests, eliminating the silent regression and reducing detection gap.

Рабочий процесс

• 1. Detect ticket spike and reproduce locally. 2. Replay last 500 documents to quantify failure rate. 3. Revert prompt to previous version and confirm rate drops. 4. Backfill affected records with pinned prompt and compare totals. 5. Version prompts by hashing content and logging hash with each extraction. 6. Refactor parser to use strict Pydantic models and arithmetic checks. 7. Create golden file tests that call the model and assert expected structure. 8. Gate merges on passing golden file tests.

Доказательства

First, someone edited the extraction prompt in place to make output "cleaner", with no version pin and no changelog entry, so the diff never appeared in a code review.

Источник

The downstream parser treated missing fields as optional and substituted defaults instead of failing loudly, which converted a hard error into a soft data corruption.

Источник

The team had unit tests for the parser, but every fixture was hand‑written valid JSON, so the tests proved the parser worked on the input it was designed for.

Источник

Reverting the prompt is a repair, not a fix. The durable version has three parts that reinforce each other: version every prompt and model reference as data rather than as a string in a function, make the parser strict so malformed output fails loudly in staging, and gate merges on a golden set that actually calls a model.

Источник