I Built an Agent Eval Harness. Real Agents Broke the Clean Version of the Story

Description and use

AgentEval Forge is an evaluation harness that shows integration and adapter realism are more critical than simple metrics. With only 9 % pass rate, the field test revealed that most failures stem from incompatibility rather than agent quality.

Problem

Evaluating real-world LLM agents requires more than scoring; it demands robust integration, adapter realism, and safety enforcement.

Outcome

AgentEval Forge successfully imports, configures, and scores real third‑party agents at scale, revealing adapter realism as the primary bottleneck and demonstrating the necessity of field testing for realistic evaluation.

Workflow

• Define 20 critical user journeys and 118 tasks in the PRD
• Implement five core components: scenario pack engine, runner, scoring layer, regression engine, adversarial generator
• Create adapters for five agent surfaces with restricted invocation payloads
• Build security model with sandbox, trust policies, audit trail, API key sanitization
• Integrate CI via GitHub Actions, GitLab CI, Docker sandbox
• Document user guide, scoring guide, scenario authoring guide, and field test reports
• Conduct field testing: search 150+ repos, select 19 viable agents, bucket by star count
• Configure each agent with its own venv, scenario pack, and adapter
• Run local, cheap, and better tiers using Qwen3.5‑9B‑MLX‑4bit, gpt‑4o‑mini, and gpt‑4o
• Collect pass/fail data, identify adapter and configuration issues
• Iterate adapters to fix hardcoded models, path writes, and packaging problems
• Publish results and lessons learned

Evidence

The pass rate was 9%. Across all 19 agents and 95 scenarios, only 9 passes.

Source

Real agents turned into the tsunami I was not prepared for, and the project became an integration reality check much faster than I expected.

Source