Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

I Built an Agent Eval Harness. Real Agents Broke the Clean Version of the Story

说明与用途

AgentEval Forge 是一个评估工具,展示了集成和适配器的真实性比简单指标更重要。仅 9% 的通过率表明大多数失败源于不兼容,而非代理质量。

问题

Evaluating real-world LLM agents requires more than scoring; it demands robust integration, adapter realism, and safety enforcement.

结果

AgentEval Forge successfully imports, configures, and scores real third‑party agents at scale, revealing adapter realism as the primary bottleneck and demonstrating the necessity of field testing for realistic evaluation.

工作流

  • Define 20 critical user journeys and 118 tasks in the PRD
  • Implement five core components: scenario pack engine, runner, scoring layer, regression engine, adversarial generator
  • Create adapters for five agent surfaces with restricted invocation payloads
  • Build security model with sandbox, trust policies, audit trail, API key sanitization
  • Integrate CI via GitHub Actions, GitLab CI, Docker sandbox
  • Document user guide, scoring guide, scenario authoring guide, and field test reports
  • Conduct field testing: search 150+ repos, select 19 viable agents, bucket by star count
  • Configure each agent with its own venv, scenario pack, and adapter
  • Run local, cheap, and better tiers using Qwen3.5‑9B‑MLX‑4bit, gpt‑4o‑mini, and gpt‑4o
  • Collect pass/fail data, identify adapter and configuration issues
  • Iterate adapters to fix hardcoded models, path writes, and packaging problems
  • Publish results and lessons learned

证据

The pass rate was 9%. Across all 19 agents and 95 scenarios, only 9 passes.

来源

Real agents turned into the tsunami I was not prepared for, and the project became an integration reality check much faster than I expected.

来源

  • 记录日期: 2026-09-08T03:07:45.193Z