We Benchmarked 5 OpenClaw Skill Scanners. Recall Went From 8% to 95%

Описание и применение

trentclaw, агент‑основанный сканер, выявил 94,6 % опасных навыков, превзойдя остальные инструменты, что подтверждает важность логического анализа навыков.

Задача

ClawHub hosts over 60K skills, many vulnerable or malicious; existing scanners miss many dangerous skills, especially those that ship no code.

Результат

trentclaw outperformed all other scanners, achieving 94.6% recall and 88% precision, highlighting the importance of agent-based reasoning for skill auditing.

Рабочий процесс

• 1. Curated a benchmark set of 60 OpenClaw skills, manually labeled as benign, vulnerable, or malicious.
• 2. Collapsed vulnerable and malicious into a single flagged class for comparison.
• 3. Ran each of the five scanners on the 54-skill intersection they could process.
• 4. Collected accuracy, precision, recall, and F1 metrics for each scanner.
• 5. Analyzed overlap and unique findings among scanners.
• 6. Identified that agent-based reasoning (trentclaw) audits SKILL.md and fetch instructions to catch prose-only attacks.
• 7. Documented results and provided full methodology on the benchmark blog.

Доказательства

The agent-based scanner (trentclaw) caught 94.6% of potentially dangerous skills, the only scanner above 60%.

Источник

Static analysis is high precision / very low recall. trentclaw lands around 94.6% recall and 88% precision.

Источник