The Model Changed. My Skill Didn't. The Score Still Dropped.

说明与用途

模型更新后评估分数下降，但技能未变。作者发现问题在于评估工具的变化，而非技能本身，并将技能稳定在最低支持层级，同时选择更稳定的评估模型，提升了可靠性。

问题

A regression in evaluation scores was observed after a model update, but the skill itself had not changed.

结果

By recording instrument details and regrading failures, the author discovered that score regressions were due to changes in the judge and model, not the skill. The skill was stabilized on the weakest supported tier and the judge was switched to a steadier model, leading to improved and reliable evaluation results.

工作流

• Record instrument details (resolved model IDs, judge‑prompt hash, CLI version, session shape) for each run.
• Compare instrument changes before interpreting score changes.
• Regrade stored failures with the same transcripts to isolate judge variance.
• Validate expectation texts against the skill’s actual requirements.
• Adjust the skill only when a failure is confirmed by the judge and the expectation text.
• Choose the cheapest judge that is steady enough for the suite.
• Maintain a series gate and keep failed measurements for future analysis.

证据

The alias resolved to Claude Sonnet 5. Three full runs scored 100 · 100 · 98 of 101.

来源

A few days later the 0.18.2 tag measured 92 · 96 · 93 of 101. The alias now resolved to Sonnet 5.5, for agent and judge.

来源

Regrade before reword, and check the expectation text before either.

来源

The judge prompt stayed untouched across the switch, same hash, only the model changed.

来源