Eval-Driven Vibe Coding: Implementing AI Evals with Golden Cases

说明与用途

作者介绍了一种名为“评估驱动的氛围编码”的AI代理测试工作流。该方法不使用模拟对象，而是针对由真实故障衍生出的20-50个“黄金”测试用例运行真实模型。这能捕捉到标准单元测试无法发现的行为回归（如工具调用）。完整评估套件每次运行成本约为0.56美元。

问题

Mocked unit tests fail to detect regressions in AI agent behavior (e.g., tool calling logic) because they mock away the model's actual output. Manual 'vibe checking' is inconsistent and misses subtle 'almost right' failures, leading to silent regressions in production.

结果

The author established a suite of 36 golden cases with zero new dependencies. The cost to run the full eval suite is approximately $0.56 per run. This approach catches behavioral regressions that mocked tests miss, without requiring a large ML team or expensive research benchmarks.

工作流

• Ship the AI feature behind standard build-time quality gates (tests, types, review).
• Monitor production traces and support threads to identify real failures.
• Encode each identified failure as a 'golden case' in a JSONL file, defining the input and expected output (e.g., specific tool calls).
• Integrate these golden cases into the CI pipeline as a regression tier to prevent the specific failure from recurring.
• Keep the LLM judge binary for gating and avoid letting it block CI entirely; use it for reporting on non-deterministic aspects.

证据

A full product eval suite costs about $0.56 a run

来源

89% of teams building AI agents have observability wired up. Barely half run offline evals against a test set

来源

20-50 simple tasks drawn from real failures is a great start

来源