The Model-Change Protocol: What I Actually Do When the Tally Shows Two Weeks of Model Drift

Description and use

A five‑step protocol detects and resolves model drift, preventing unnecessary model swaps and parameter tuning.

Problem

Two consecutive weeks of model-tag growth indicated a model problem, but prior investigations had misattributed drift to model changes instead of prompt edits.

Outcome

The protocol resolved the drift without swapping models, saved ~2.5 hours, updated the contract by one line, re‑baselined two fixtures, and established a revisit schedule.

Workflow

• Declare a model incident and freeze version bumps
• Run the golden fixture suite against both behaviors
• Diff the divergences, not the averages
• Take one of three decisions (pin, adopt, quarantine)
• Log the decision with an expiry date

Evidence

The protocol exists to make sure I never do that again.

Source

Step 1 — Declare a model incident and freeze version bumps.

Source

Step 2 — Run the golden fixture suite against both behaviors.

Source

Step 3 — Diff the divergences, not the averages.

Source

Step 4 — Take one of three decisions.

Source

Step 5 — Log the decision with an expiry date.

Source

Two weeks of tallies, one incident, one evening, 2.5 hours, one contract line, two fixtures re-baselined, zero model swaps.

Source