The Model-Change Protocol: What I Actually Do When the Tally Shows Two Weeks of Model Drift

说明与用途

五步协议帮助检测并解决模型漂移，避免不必要的模型切换和参数调优。

问题

Two consecutive weeks of model-tag growth indicated a model problem, but prior investigations had misattributed drift to model changes instead of prompt edits.

结果

The protocol resolved the drift without swapping models, saved ~2.5 hours, updated the contract by one line, re‑baselined two fixtures, and established a revisit schedule.

工作流

• Declare a model incident and freeze version bumps
• Run the golden fixture suite against both behaviors
• Diff the divergences, not the averages
• Take one of three decisions (pin, adopt, quarantine)
• Log the decision with an expiry date

证据

The protocol exists to make sure I never do that again.

来源

Step 1 — Declare a model incident and freeze version bumps.

来源

Step 2 — Run the golden fixture suite against both behaviors.

来源

Step 3 — Diff the divergences, not the averages.

来源

Step 4 — Take one of three decisions.

来源

Step 5 — Log the decision with an expiry date.

来源

Two weeks of tallies, one incident, one evening, 2.5 hours, one contract line, two fixtures re-baselined, zero model swaps.

来源