We scanned nine popular AI agent repos. Here is what an "evidence-led registry" taught us
说明与用途
作者创建了 Metal Mantra,一个自动化注册表,扫描 AI 代理仓库并生成跨五个维度的详细报告。对九个流行仓库的评分为 69–84/100,揭示了缺失的安全防护和未知因素。关键经验是:文件选择策略至关重要,错误的采样会导致误判。
问题
Assessing AI agent repositories requires objective, reproducible metrics rather than subjective popularity claims.
结果
Nine well-known agent repos scored 69–84/100; the scanner revealed missing guardrails, loose schemas, and unbounded loops, and highlighted sampling strategy as a key factor in accurate assessment.
工作流
- 1. Submit a public GitHub repo and official domain to Metal Mantra.
- 2. The scanner reads eligible files (up to 80) in archive order, applying deterministic rules.
- 3. Rules produce deductions linked to specific files and a weighted score across five pillars.
- 4. The report lists unknowns, partial selections, and a "what this does not establish" disclaimer.
- 5. Users can review the report, suggest rule adjustments, and re-run scans.
- 6. For private repos, owners run the scanner in GitHub Actions, sending only rule IDs and counts via OIDC for CI-attested reports.
证据
Most AI agent directories answer one question: who pitched best? If an agent is going to act for you (book things, read your files, call your tools), that is the wrong question.
You paste a public GitHub repository and an official domain. Deterministic rules read the source and produce a public report across five weighted pillars.
For our first scans we read the first 80 eligible files in archive order. For a monorepo like an agents SDK, those 80 files were examples and docs. The scanner was effectively grading the framework by its demos, and it failed the "is this an agent?" check on one of the best-known agent SDKs in existence.
- 记录日期: 2026-10-09T17:41:09.920Z