Your SKILL.md is production config. Test it like one.

Description and use

skilldiff tests agent skill changes by running them twice, capturing actual behavior, and asserting against expected outcomes, catching regressions missed by diffs and offline evals.

Problem

Edits to SKILL.md files can silently change agent behavior, causing regressions that are hard to detect until weeks later.

Outcome

skilldiff enables deterministic, reproducible detection of behavioral regressions in agent skills, reducing late-stage bugs and improving PR quality.

Workflow

• 1. Run npx skilldiff init to discover skills and generate starter scenarios.
• 2. Define assertions in a YAML scenario file (e.g., files_changed, tool_calls, must_not).
• 3. Execute npx skilldiff run <scenario> --live --base origin/main for live testing or CI mode for deterministic runs.
• 4. Review the behavior report in the PR comment; failures block the merge.

Evidence

A text diff would never have caught that. That's behavioral regression.

Source

For CI it replays recorded traces, so it's deterministic and needs no API key or credits.

Source