I wrote evals for my Claude Code plugin. The first run proved my README wrong.

Description and use

A Claude Code plugin that auto‑generates harnesses from a project description initially failed to trigger on bare descriptions. After iterative evals, trigger widening, grader fixes, and model‑specific updates, it now passes all tests and has improved guard coverage.

Problem

The plugin’s README claimed it would start setup from a bare project description, but the first eval run showed the skill never fired on such input.

Outcome

After iterative evals and fixes, the plugin now correctly triggers on bare descriptions, passes all eight eval cases on the latest model, and has improved guard coverage from 0/13 to 13/13 when counting permission rules.

Workflow

• 1. Built a plugin that generates harnesses from a one‑paragraph description.
• 2. Created an eval suite of eight cases covering key plugin claims.
• 3. Ran the eval suite; identified failures in trigger, graders, and a Claude Code update regression.
• 4. Widened the trigger to accept bare descriptions.
• 5. Rewrote graders to use deterministic text matches where possible.
• 6. Added file existence checks before finalizing the harness.
• 7. Fixed model‑specific trigger and added new cases for smaller models.
• 8. Ran the updated plugin on all cases, achieving 8/8 on the latest model and improved guard coverage.
• 9. Developed an attack suite to test guard scripts directly, revealing missing guard coverage and spec issues.
• 10. Updated the plugin spec and documentation accordingly.

Evidence

The README said you could just describe your project and the setup would start.

Source

The first eval run said otherwise. Given only a project description, with no mention of Claude Code or setting anything up, the skill never fired.

Source

I widened the trigger, and self-triggers-on-bare-description now makes sure it stays fixed.

Source

In October, on a newer Claude Code version, vague-description failed twice in a row.

Source

After fixing the trigger and adding four new cases, version 0.7.0 passes 9 of 12 on Haiku and 9 of 12 on Sonnet.

Source