My skill passed every structural check. Then it scored 7/12.

Описание и применение

Скилл, проверенный на структурные критерии, изначально ошибочно классифицировал границы, но после пересмотра логики достиг 100 % точности на тестах.

Задача

A production audit skill failed to correctly classify borderline cases, causing over‑escalation and incorrect readiness verdicts.

Результат

The revised skill achieved perfect scores on both blind and independent test suites, eliminating over‑escalation and confirming correct calibration.

Рабочий процесс

• Extract behavior spec from skill text
• Detect execution mode
• Generate blind cases covering core behavior, calibration boundaries, adversarial safety, edge inputs, and a clean control
• Freeze expectations before execution
• Run skill on fresh runners with only skill text and case input
• Grade archived outputs with deterministic grader and safety sub‑scores
• Cluster failures into root causes
• Revise decision logic and re‑grade

Доказательства

Result: 7/12.

Источник

The skill caught every planted defect — detection was perfect.

Источник

The revision changed one step: a calibration ladder with explicit precedence.

Источник

Same blind suite, same fixtures: 12/12, with the injection refused and flagged, the API key redacted, and the URL claims honestly labeled "not checked."

Источник

A later, independent 13-case suite reproduced the differential on identical fixtures: the old version scored 10/13 — gate NOT MET — and the revised version 13/13, gate MET, safety sub-scores 0/0/0/0.

Источник