I measured whether my Claude skills help. One of them made the code worse

Описание и применение

Автоматизированный тест skill-be показал, что навыки Claude значительно улучшают качество кода, но один навык ухудшил критерий, выявив отсутствие примера использования matchAsync.

Задача

Determine if adding a skill to Claude improves code quality and identify any negative impact.

Результат

The experiment showed a dramatic average improvement from 22% to 96% across 139 criteria. One skill (validate) caused a specific criterion to worsen, revealing a missing example for using matchAsync to convert a Validate into a response. Adding the example fixed the issue, raising all related criteria to 100%.

Рабочий процесс

• 1. Define scenarios and criteria for each skill.
• 2. Run each scenario twice: baseline (no skill) and with SKILL.md prepended.
• 3. Execute each run 3 times to mitigate randomness.
• 4. Use a second LLM (Claude Haiku 5.5) as grader to evaluate each answer against criteria, grading each answer 3 times and taking majority.
• 5. Aggregate scores per criterion and apply rules to flag NOT READY skills.
• 6. For failing criteria, invoke skill-be triage to request fresh answers and propose causes.
• 7. Update SKILL.md with missing steps or examples, then re-run to verify improvement.

Доказательства

The experiment is the simplest one there is: change one thing and compare.

Average criterion score without the skill 22%

Average criterion score with the skill 96%

Criteria the skill made worse 1

The skill made one thing worse