We Benchmarked 5 OpenClaw Skill Scanners. Recall Went From 8% to 95%

说明与用途

trentclaw，基于代理的扫描器，识别了94.6%的危险技能，优于其他工具，强调了逻辑技能分析的重要性。

问题

ClawHub hosts over 60K skills, many vulnerable or malicious; existing scanners miss many dangerous skills, especially those that ship no code.

结果

trentclaw outperformed all other scanners, achieving 94.6% recall and 88% precision, highlighting the importance of agent-based reasoning for skill auditing.

工作流

• 1. Curated a benchmark set of 60 OpenClaw skills, manually labeled as benign, vulnerable, or malicious.
• 2. Collapsed vulnerable and malicious into a single flagged class for comparison.
• 3. Ran each of the five scanners on the 54-skill intersection they could process.
• 4. Collected accuracy, precision, recall, and F1 metrics for each scanner.
• 5. Analyzed overlap and unique findings among scanners.
• 6. Identified that agent-based reasoning (trentclaw) audits SKILL.md and fetch instructions to catch prose-only attacks.
• 7. Documented results and provided full methodology on the benchmark blog.

证据

The agent-based scanner (trentclaw) caught 94.6% of potentially dangerous skills, the only scanner above 60%.

来源

Static analysis is high precision / very low recall. trentclaw lands around 94.6% recall and 88% precision.

来源