From SkillCheck to TraceMantle: Tracking Agent Skill Changes and Test Evidence

Description and use

TraceMantle extends SkillCheck by tracking all skill package files, linking test evidence to package changes, and automating evidence comparison, enabling developers to quickly identify which tests need rerunning.

Problem

Developers needed a tool that not only validates AI agent skill files but also tracks changes in supporting resources and determines whether existing test evidence remains applicable.

Outcome

TraceMantle extends SkillCheck by tracking all skill package files, linking test evidence to package changes, and automating evidence comparison, enabling developers to quickly identify which tests need rerunning and ensuring evidence remains trustworthy.

Workflow

• 1. Run TraceMantle on the skill package to validate frontmatter, references, size limits, and compatibility.
• 2. Generate a bundle manifest that fingerprints all files in the package.
• 3. Import test evidence from a Promptfoo export (version‑pinned).
• 4. Compare the imported evidence against the current package and a trusted policy to identify changed inputs, incompatible results, and checks requiring rerun.
• 5. Review TraceMantle output (text, JSON, or GitHub annotations) and decide whether to approve evidence or rerun tests.

Evidence

TraceMantle retains that functionality and adds support for tracking the files in a skill package, comparing changes, and checking whether previous test evidence still applies.

Source

The expanded functionality adds three related capabilities: Bundle manifests, Evidence imports, Evidence comparison.

Source