My skill passed every structural check. Then it scored 7/12.

Description and use

A skill that passed structural checks initially misclassified borderline cases; after revising its decision logic, it achieved perfect scores on blind and independent test suites.

Problem

A production audit skill failed to correctly classify borderline cases, causing over‑escalation and incorrect readiness verdicts.

Outcome

The revised skill achieved perfect scores on both blind and independent test suites, eliminating over‑escalation and confirming correct calibration.

Workflow

• Extract behavior spec from skill text
• Detect execution mode
• Generate blind cases covering core behavior, calibration boundaries, adversarial safety, edge inputs, and a clean control
• Freeze expectations before execution
• Run skill on fresh runners with only skill text and case input
• Grade archived outputs with deterministic grader and safety sub‑scores
• Cluster failures into root causes
• Revise decision logic and re‑grade

Evidence

Result: 7/12.

Source

The skill caught every planted defect — detection was perfect.

Source

The revision changed one step: a calibration ladder with explicit precedence.

Source

Same blind suite, same fixtures: 12/12, with the injection refused and flagged, the API key redacted, and the URL claims honestly labeled "not checked."

Source

A later, independent 13-case suite reproduced the differential on identical fixtures: the old version scored 10/13 — gate NOT MET — and the revised version 13/13, gate MET, safety sub-scores 0/0/0/0.

Source