A Model Swap Is a Migration, Not a Toggle

Описание и применение

Замена модели в продакшене – это миграция, а не переключатель. Использование фикстур и теневого запуска позволяет обнаружить несоответствия схемы и сдвиг текста до развертывания.

Задача

Changing a single environment variable to swap the provider for the support‑reply feature caused data inconsistencies and missing fields in exported data, revealing that a model swap is a schema‑affecting migration rather than a simple toggle.

Результат

Implementing a fixture‑based harness for provider swaps prevents schema drift, ensures data integrity, and provides a clear gate for deployment.

Рабочий процесс

• Capture real requests and baseline outputs into JSON fixtures with redaction.
• Run a shadow process that replays each fixture through the candidate provider, checking schema compliance and text overlap.
• Collect shape errors, token overlap, and latency metrics for each fixture.
• Review the fixture artifacts in a pull request to ensure no data drift or schema violations.
• Gate the deployment until all fixtures pass validation.
• Deploy the new provider configuration.
• Monitor exports for missing or altered fields.
• Rollback if new provider causes schema or data issues.

Доказательства

I changed a single environment variable, pointed the support‑reply feature at a different provider, and went home feeling clever.

Источник

On Monday the export job had three days of rows where refund_days was null, and nobody flagged it because the UI rendered the reply text and quietly ignored the missing field.

Источник

The cheapest honest evidence is a frozen fixture corpus captured from production before you touch the provider.

Источник

Capture writes one JSON file per real request, keeping the input and the baseline output so later runs have something to compare against.

Источник

The shadow run replays every fixture through the candidate and scores two things independently: whether the payload still parses into the shape the database expects, and how far the text drifted from the baseline.

Источник