Do AI agent skills actually improve results? How I test them, including the ones that failed

Описание и применение

Проверка, повышают ли навыки агента Claude качество ответов, показала, что некоторые навыки улучшают результаты, другие не дают прироста, а один ухудшает их.

Задача

Determine whether adding a skill file to a Claude agent improves answer quality on tasks it was not tuned for.

Результат

MCP Mechanic and several other skills improved combined scores; Blueprint Probe did not improve flaw detection; Guide Proof 1.0.0 worsened results; Guide Proof 1.1.0 improved but still below baseline on one task.

Рабочий процесс

• Write tasks based on skill public description only
• Freeze tasks, fixtures, rubrics and answer keys into FREEZE.txt
• Run baseline and skill conditions in separate temp dirs
• Copy skill folder into .claude/skills/ for skill condition, excluding evals
• Execute claude -p headless with file tools only
• Collect answers and run deterministic hard checks
• Score with blind rubric via separate grader model
• Combine 0.5×hard-check pass rate + 0.5×mean rubric score
• Compare baseline vs skill results

Доказательства

The script's own method note explains the catch: the checks "test whether named issues and safe behaviours appear, not overall answer quality."

Источник

The grader runs with no tools. In the generic mode it scores each rubric item 0, 0.5 or 1, and it lists any claim in the answer it believes is factually wrong.

Источник

MCP Mechanic (first round) baseline 0.876, with skill 0.976, 15 runs per condition.

Источник

Blueprint Probe reviews automation designs. Planted‑flaw recall was 1.00 in both conditions.

Источник