Eval-Driven Vibe Coding: Implementing AI Evals with Golden Cases

Описание и применение

Автор описывает метод «eval-driven vibe coding» для тестирования AI-агентов. Вместо моков используются реальные модели и набор из 20-50 «золотых» тестов, созданных на основе реальных ошибок. Это позволяет обнаруживать регрессии в поведении модели (например, вызов инструментов), которые пропускают стандартные юнит-тесты. Стоимость запуска полного набора тестов составляет около $0.56.

Задача

Mocked unit tests fail to detect regressions in AI agent behavior (e.g., tool calling logic) because they mock away the model's actual output. Manual 'vibe checking' is inconsistent and misses subtle 'almost right' failures, leading to silent regressions in production.

Результат

The author established a suite of 36 golden cases with zero new dependencies. The cost to run the full eval suite is approximately $0.56 per run. This approach catches behavioral regressions that mocked tests miss, without requiring a large ML team or expensive research benchmarks.

Рабочий процесс

• Ship the AI feature behind standard build-time quality gates (tests, types, review).
• Monitor production traces and support threads to identify real failures.
• Encode each identified failure as a 'golden case' in a JSONL file, defining the input and expected output (e.g., specific tool calls).
• Integrate these golden cases into the CI pipeline as a regression tier to prevent the specific failure from recurring.
• Keep the LLM judge binary for gating and avoid letting it block CI entirely; use it for reporting on non-deterministic aspects.

Доказательства

A full product eval suite costs about $0.56 a run

Источник

89% of teams building AI agents have observability wired up. Barely half run offline evals against a test set

Источник

20-50 simple tasks drawn from real failures is a great start

Источник