Do AI agent skills actually improve results? How I test them, including the ones that failed

说明与用途

测试向Claude代理添加技能是否能提升答案质量，结果显示部分技能提升了效果，部分无效，另一个则降低了质量。

问题

Determine whether adding a skill file to a Claude agent improves answer quality on tasks it was not tuned for.

结果

MCP Mechanic and several other skills improved combined scores; Blueprint Probe did not improve flaw detection; Guide Proof 1.0.0 worsened results; Guide Proof 1.1.0 improved but still below baseline on one task.

工作流

• Write tasks based on skill public description only
• Freeze tasks, fixtures, rubrics and answer keys into FREEZE.txt
• Run baseline and skill conditions in separate temp dirs
• Copy skill folder into .claude/skills/ for skill condition, excluding evals
• Execute claude -p headless with file tools only
• Collect answers and run deterministic hard checks
• Score with blind rubric via separate grader model
• Combine 0.5×hard-check pass rate + 0.5×mean rubric score
• Compare baseline vs skill results

证据

The script's own method note explains the catch: the checks "test whether named issues and safe behaviours appear, not overall answer quality."

来源

The grader runs with no tools. In the generic mode it scores each rubric item 0, 0.5 or 1, and it lists any claim in the answer it believes is factually wrong.

来源

MCP Mechanic (first round) baseline 0.876, with skill 0.976, 15 runs per condition.

来源

Blueprint Probe reviews automation designs. Planted‑flaw recall was 1.00 in both conditions.

来源