My skill passed every structural check. Then it scored 7/12.

说明与用途

一款通过结构检查的技能最初错误分类边界案例，经过逻辑修订后在盲测和独立验证中均达成100%准确率。

问题

A production audit skill failed to correctly classify borderline cases, causing over‑escalation and incorrect readiness verdicts.

结果

The revised skill achieved perfect scores on both blind and independent test suites, eliminating over‑escalation and confirming correct calibration.

工作流

• Extract behavior spec from skill text
• Detect execution mode
• Generate blind cases covering core behavior, calibration boundaries, adversarial safety, edge inputs, and a clean control
• Freeze expectations before execution
• Run skill on fresh runners with only skill text and case input
• Grade archived outputs with deterministic grader and safety sub‑scores
• Cluster failures into root causes
• Revise decision logic and re‑grade

证据

Result: 7/12.

来源

The skill caught every planted defect — detection was perfect.

来源

The revision changed one step: a calibration ladder with explicit precedence.

来源

Same blind suite, same fixtures: 12/12, with the injection refused and flagged, the API key redacted, and the URL claims honestly labeled "not checked."

来源

A later, independent 13-case suite reproduced the differential on identical fixtures: the old version scored 10/13 — gate NOT MET — and the revised version 13/13, gate MET, safety sub-scores 0/0/0/0.

来源