I measured whether my Claude skills help. One of them made the code worse

Description and use

Automated testing with skill-be revealed that Claude skills substantially improve code quality, yet one skill worsened a criterion due to a missing matchAsync example.

Problem

Determine if adding a skill to Claude improves code quality and identify any negative impact.

Outcome

The experiment showed a dramatic average improvement from 22% to 96% across 139 criteria. One skill (validate) caused a specific criterion to worsen, revealing a missing example for using matchAsync to convert a Validate into a response. Adding the example fixed the issue, raising all related criteria to 100%.

Workflow

• 1. Define scenarios and criteria for each skill.
• 2. Run each scenario twice: baseline (no skill) and with SKILL.md prepended.
• 3. Execute each run 3 times to mitigate randomness.
• 4. Use a second LLM (Claude Haiku 5.5) as grader to evaluate each answer against criteria, grading each answer 3 times and taking majority.
• 5. Aggregate scores per criterion and apply rules to flag NOT READY skills.
• 6. For failing criteria, invoke skill-be triage to request fresh answers and propose causes.
• 7. Update SKILL.md with missing steps or examples, then re-run to verify improvement.

Evidence

The experiment is the simplest one there is: change one thing and compare.

Average criterion score without the skill 22%

Average criterion score with the skill 96%

Criteria the skill made worse 1

The skill made one thing worse