The Model Changed. My Skill Didn't. The Score Still Dropped.
Description and use
A regression in evaluation scores after a model update was traced to changes in the evaluation instrument, not the skill. The skill was stabilized on the weakest supported tier and a steadier judge model was chosen, improving reliability.
Problem
A regression in evaluation scores was observed after a model update, but the skill itself had not changed.
Outcome
By recording instrument details and regrading failures, the author discovered that score regressions were due to changes in the judge and model, not the skill. The skill was stabilized on the weakest supported tier and the judge was switched to a steadier model, leading to improved and reliable evaluation results.
Workflow
- Record instrument details (resolved model IDs, judge‑prompt hash, CLI version, session shape) for each run.
- Compare instrument changes before interpreting score changes.
- Regrade stored failures with the same transcripts to isolate judge variance.
- Validate expectation texts against the skill’s actual requirements.
- Adjust the skill only when a failure is confirmed by the judge and the expectation text.
- Choose the cheapest judge that is steady enough for the suite.
- Maintain a series gate and keep failed measurements for future analysis.
Evidence
The alias resolved to Claude Sonnet 5. Three full runs scored 100 · 100 · 98 of 101.
A few days later the 0.18.2 tag measured 92 · 96 · 93 of 101. The alias now resolved to Sonnet 5.5, for agent and judge.
Regrade before reword, and check the expectation text before either.
The judge prompt stayed untouched across the switch, same hash, only the model changed.
- Record date: 2026-10-07T10:40:57.466Z