We scanned nine popular AI agent repos. Here is what an "evidence-led registry" taught us
Описание и применение
Автор создал Metal Mantra – автоматический реестр, который сканирует репозитории AI‑агентов, выдавая детальные отчёты по пяти критериям. На девяти популярных репозиториях оценки колебались от 69 до 84 баллов, выявляя недостающие защитные механизмы и неопределённости. Ключевой урок – выбор файлов для анализа критичен: неправильная стратегия выборки может исказить результаты.
Задача
Assessing AI agent repositories requires objective, reproducible metrics rather than subjective popularity claims.
Результат
Nine well-known agent repos scored 69–84/100; the scanner revealed missing guardrails, loose schemas, and unbounded loops, and highlighted sampling strategy as a key factor in accurate assessment.
Рабочий процесс
- 1. Submit a public GitHub repo and official domain to Metal Mantra.
- 2. The scanner reads eligible files (up to 80) in archive order, applying deterministic rules.
- 3. Rules produce deductions linked to specific files and a weighted score across five pillars.
- 4. The report lists unknowns, partial selections, and a "what this does not establish" disclaimer.
- 5. Users can review the report, suggest rule adjustments, and re-run scans.
- 6. For private repos, owners run the scanner in GitHub Actions, sending only rule IDs and counts via OIDC for CI-attested reports.
Доказательства
Most AI agent directories answer one question: who pitched best? If an agent is going to act for you (book things, read your files, call your tools), that is the wrong question.
You paste a public GitHub repository and an official domain. Deterministic rules read the source and produce a public report across five weighted pillars.
For our first scans we read the first 80 eligible files in archive order. For a monorepo like an agents SDK, those 80 files were examples and docs. The scanner was effectively grading the framework by its demos, and it failed the "is this an agent?" check on one of the best-known agent SDKs in existence.
- Дата записи: 2026-10-09T17:41:09.920Z