I Built an AI to Judge AI Projects. Here’s What It Looks For

Description and use

AI‑Atlas evaluates GitHub repositories by inspecting code, experiments, and documentation, offering transparent, evidence‑based quality scores instead of relying on superficial labels.

Problem

GitHub repositories often label themselves with generic tags like "Transformer" or "Computer Vision" but this does not reveal the true technical depth, originality, or reproducibility of the work.

Outcome

AI‑Atlas provides evidence‑grounded, transparent scores that distinguish between superficial labeling and substantive technical contribution, enabling developers and recruiters to assess AI projects more accurately.

Workflow

• Discover repositories via public GitHub APIs
• Classify projects into six domains (ML, CV, DS, NLP & LLMs, AI Agents, MLOps)
• Parse code to identify implementation details, training loops, configs, and outputs
• Evaluate each project against the seven rubric dimensions using evidence from files
• Generate a detailed judgment report with reasoning and supporting evidence
• Compute a Project Merit score via versioned scoring rules, separating interpretation from scoring

Evidence

A GitHub repository can say "Transformer," "RAG," or "Computer Vision" in the README. That does not tell you whether the work is technically deep, original, reproducible, or even really yours.

Source

AI‑Atlas currently maps projects across Machine Learning, Computer Vision, Data Science, NLP & LLMs, AI Agents, and MLOps, but the domain classification itself is only the beginning.

Source

Each relevant repository is examined across seven dimensions: Technical Depth, Conceptual Difficulty, Originality/Modification, Completeness, Practical/Research Usefulness, Documentation & Educational Value, and Engineering & Reproducibility.

Source

AI‑Atlas traces conclusions back toward implementation, training, and evaluation evidence rather than relying only on repository descriptions.

Source