We Benchmarked 5 OpenClaw Skill Scanners. Recall Went From 8% to 95%

Description and use

trentclaw, an agent-based scanner, identified 94.6% of dangerous skills, outperforming other tools and underscoring the value of logical skill analysis.

Problem

ClawHub hosts over 60K skills, many vulnerable or malicious; existing scanners miss many dangerous skills, especially those that ship no code.

Outcome

trentclaw outperformed all other scanners, achieving 94.6% recall and 88% precision, highlighting the importance of agent-based reasoning for skill auditing.

Workflow

• 1. Curated a benchmark set of 60 OpenClaw skills, manually labeled as benign, vulnerable, or malicious.
• 2. Collapsed vulnerable and malicious into a single flagged class for comparison.
• 3. Ran each of the five scanners on the 54-skill intersection they could process.
• 4. Collected accuracy, precision, recall, and F1 metrics for each scanner.
• 5. Analyzed overlap and unique findings among scanners.
• 6. Identified that agent-based reasoning (trentclaw) audits SKILL.md and fetch instructions to catch prose-only attacks.
• 7. Documented results and provided full methodology on the benchmark blog.

Evidence

The agent-based scanner (trentclaw) caught 94.6% of potentially dangerous skills, the only scanner above 60%.

Source

Static analysis is high precision / very low recall. trentclaw lands around 94.6% recall and 88% precision.

Source