I wrote evals for my Claude Code plugin. The first run proved my README wrong.

Описание и применение

Плагин Claude Code, генерирующий наборы для проектов из описания, изначально не запускался при простом описании. После серии тестов и исправлений, включая расширение триггера, доработку оценщиков и проверку на обновления платформы, плагин теперь корректно запускается и проходит все тесты, а защита от атак улучшена.

Задача

The plugin’s README claimed it would start setup from a bare project description, but the first eval run showed the skill never fired on such input.

Результат

After iterative evals and fixes, the plugin now correctly triggers on bare descriptions, passes all eight eval cases on the latest model, and has improved guard coverage from 0/13 to 13/13 when counting permission rules.

Рабочий процесс

• 1. Built a plugin that generates harnesses from a one‑paragraph description.
• 2. Created an eval suite of eight cases covering key plugin claims.
• 3. Ran the eval suite; identified failures in trigger, graders, and a Claude Code update regression.
• 4. Widened the trigger to accept bare descriptions.
• 5. Rewrote graders to use deterministic text matches where possible.
• 6. Added file existence checks before finalizing the harness.
• 7. Fixed model‑specific trigger and added new cases for smaller models.
• 8. Ran the updated plugin on all cases, achieving 8/8 on the latest model and improved guard coverage.
• 9. Developed an attack suite to test guard scripts directly, revealing missing guard coverage and spec issues.
• 10. Updated the plugin spec and documentation accordingly.

Доказательства

The README said you could just describe your project and the setup would start.

Источник

The first eval run said otherwise. Given only a project description, with no mention of Claude Code or setting anything up, the skill never fired.

Источник

I widened the trigger, and self-triggers-on-bare-description now makes sure it stays fixed.

Источник

In October, on a newer Claude Code version, vague-description failed twice in a row.

Источник

After fixing the trigger and adding four new cases, version 0.7.0 passes 9 of 12 on Haiku and 9 of 12 on Sonnet.

Источник