I measured whether my Claude skills help. One of them made the code worse

说明与用途

使用 skill-be 的自动化测试显示 Claude 技能显著提升代码质量，但其中一个技能因缺少 matchAsync 示例导致某个标准下降。

问题

Determine if adding a skill to Claude improves code quality and identify any negative impact.

结果

The experiment showed a dramatic average improvement from 22% to 96% across 139 criteria. One skill (validate) caused a specific criterion to worsen, revealing a missing example for using matchAsync to convert a Validate into a response. Adding the example fixed the issue, raising all related criteria to 100%.

工作流

• 1. Define scenarios and criteria for each skill.
• 2. Run each scenario twice: baseline (no skill) and with SKILL.md prepended.
• 3. Execute each run 3 times to mitigate randomness.
• 4. Use a second LLM (Claude Haiku 5.5) as grader to evaluate each answer against criteria, grading each answer 3 times and taking majority.
• 5. Aggregate scores per criterion and apply rules to flag NOT READY skills.
• 6. For failing criteria, invoke skill-be triage to request fresh answers and propose causes.
• 7. Update SKILL.md with missing steps or examples, then re-run to verify improvement.

证据

The experiment is the simplest one there is: change one thing and compare.

Average criterion score without the skill 22%

Average criterion score with the skill 96%

Criteria the skill made worse 1

The skill made one thing worse