We scanned nine popular AI agent repos. Here is what an "evidence-led registry" taught us

Description and use

The author built Metal Mantra, an automated registry that scans AI agent repositories and produces detailed reports across five pillars. Nine popular repos scored 69–84/100, revealing missing guardrails and unknowns. The key lesson: file selection strategy is critical—poor sampling can misrepresent a repository.

Problem

Assessing AI agent repositories requires objective, reproducible metrics rather than subjective popularity claims.

Outcome

Nine well-known agent repos scored 69–84/100; the scanner revealed missing guardrails, loose schemas, and unbounded loops, and highlighted sampling strategy as a key factor in accurate assessment.

Workflow

• 1. Submit a public GitHub repo and official domain to Metal Mantra.
• 2. The scanner reads eligible files (up to 80) in archive order, applying deterministic rules.
• 3. Rules produce deductions linked to specific files and a weighted score across five pillars.
• 4. The report lists unknowns, partial selections, and a "what this does not establish" disclaimer.
• 5. Users can review the report, suggest rule adjustments, and re-run scans.
• 6. For private repos, owners run the scanner in GitHub Actions, sending only rule IDs and counts via OIDC for CI-attested reports.

Evidence

Most AI agent directories answer one question: who pitched best? If an agent is going to act for you (book things, read your files, call your tools), that is the wrong question.

Source

You paste a public GitHub repository and an official domain. Deterministic rules read the source and produce a public report across five weighted pillars.

Source

For our first scans we read the first 80 eligible files in archive order. For a monorepo like an agents SDK, those 80 files were examples and docs. The scanner was effectively grading the framework by its demos, and it failed the "is this an agent?" check on one of the best-known agent SDKs in existence.

Source