I wrote evals for my Claude Code plugin. The first run proved my README wrong.

说明与用途

Claude Code 插件最初无法在仅提供项目描述时触发。通过评估、触发器扩展、评审器修正和模型更新，插件现已通过所有测试并提升了防护覆盖率。

问题

The plugin’s README claimed it would start setup from a bare project description, but the first eval run showed the skill never fired on such input.

结果

After iterative evals and fixes, the plugin now correctly triggers on bare descriptions, passes all eight eval cases on the latest model, and has improved guard coverage from 0/13 to 13/13 when counting permission rules.

工作流

• 1. Built a plugin that generates harnesses from a one‑paragraph description.
• 2. Created an eval suite of eight cases covering key plugin claims.
• 3. Ran the eval suite; identified failures in trigger, graders, and a Claude Code update regression.
• 4. Widened the trigger to accept bare descriptions.
• 5. Rewrote graders to use deterministic text matches where possible.
• 6. Added file existence checks before finalizing the harness.
• 7. Fixed model‑specific trigger and added new cases for smaller models.
• 8. Ran the updated plugin on all cases, achieving 8/8 on the latest model and improved guard coverage.
• 9. Developed an attack suite to test guard scripts directly, revealing missing guard coverage and spec issues.
• 10. Updated the plugin spec and documentation accordingly.

证据

The README said you could just describe your project and the setup would start.

来源

The first eval run said otherwise. Given only a project description, with no mention of Claude Code or setting anything up, the skill never fired.

来源

I widened the trigger, and self-triggers-on-bare-description now makes sure it stays fixed.

来源

In October, on a newer Claude Code version, vague-description failed twice in a row.

来源

After fixing the trigger and adding four new cases, version 0.7.0 passes 9 of 12 on Haiku and 9 of 12 on Sonnet.

来源