The Model Changed. My Skill Didn't. The Score Still Dropped.

Описание и применение

При обновлении модели оценка навыка упала, хотя сам код не менялся. Автор выяснил, что причина в изменении инструмента оценки, а не в навыке, и стабилизировал его на слабейшем поддерживаемом уровне, а также выбрал более надёжный судью.

Задача

A regression in evaluation scores was observed after a model update, but the skill itself had not changed.

Результат

By recording instrument details and regrading failures, the author discovered that score regressions were due to changes in the judge and model, not the skill. The skill was stabilized on the weakest supported tier and the judge was switched to a steadier model, leading to improved and reliable evaluation results.

Рабочий процесс

• Record instrument details (resolved model IDs, judge‑prompt hash, CLI version, session shape) for each run.
• Compare instrument changes before interpreting score changes.
• Regrade stored failures with the same transcripts to isolate judge variance.
• Validate expectation texts against the skill’s actual requirements.
• Adjust the skill only when a failure is confirmed by the judge and the expectation text.
• Choose the cheapest judge that is steady enough for the suite.
• Maintain a series gate and keep failed measurements for future analysis.

Доказательства

The alias resolved to Claude Sonnet 5. Three full runs scored 100 · 100 · 98 of 101.

Источник

A few days later the 0.18.2 tag measured 92 · 96 · 93 of 101. The alias now resolved to Sonnet 5.5, for agent and judge.

Источник

Regrade before reword, and check the expectation text before either.

Источник

The judge prompt stayed untouched across the switch, same hash, only the model changed.

Источник