Do AI agent skills actually improve results? How I test them, including the ones that failed

Description and use

Testing whether adding a skill to a Claude agent improves answer quality showed that some skills improve results, some do not, and one worsens them.

Problem

Determine whether adding a skill file to a Claude agent improves answer quality on tasks it was not tuned for.

Outcome

MCP Mechanic and several other skills improved combined scores; Blueprint Probe did not improve flaw detection; Guide Proof 1.0.0 worsened results; Guide Proof 1.1.0 improved but still below baseline on one task.

Workflow

• Write tasks based on skill public description only
• Freeze tasks, fixtures, rubrics and answer keys into FREEZE.txt
• Run baseline and skill conditions in separate temp dirs
• Copy skill folder into .claude/skills/ for skill condition, excluding evals
• Execute claude -p headless with file tools only
• Collect answers and run deterministic hard checks
• Score with blind rubric via separate grader model
• Combine 0.5×hard-check pass rate + 0.5×mean rubric score
• Compare baseline vs skill results

Evidence

The script's own method note explains the catch: the checks "test whether named issues and safe behaviours appear, not overall answer quality."

Source

The grader runs with no tools. In the generic mode it scores each rubric item 0, 0.5 or 1, and it lists any claim in the answer it believes is factually wrong.

Source

MCP Mechanic (first round) baseline 0.876, with skill 0.976, 15 runs per condition.

Source

Blueprint Probe reviews automation designs. Planted‑flaw recall was 1.00 in both conditions.

Source