Build a Reproducible AI Agent Evaluation Lab with Docker Compose

说明与用途

使用 Docker Compose 构建可重现的 AI 代理评估实验室，利用配置文件、固定镜像、环境清单和证据目录消除 CI 中的随机失败。

问题

Agent evaluation failures in CI are hard to diagnose because of non‑deterministic external factors such as tool responses, database state, clock, and dependency versions.

结果

The lab enables deterministic AI agent evaluation, reduces flaky CI failures, and provides detailed, reproducible evidence for debugging and regression analysis.

工作流

• 1. Define core services (e.g., fixture‑api, eval‑runner) in docker-compose.yml with image digests and fixed environment variables.
• 2. Use Compose profiles to separate optional services (fault‑proxy, otel‑collector) that can be enabled with --profile.
• 3. Pin all inputs: container images, JS dependencies, synthetic datasets, prompts, policies, tool schemas, locale, timezone, clock, seed, and provider identifiers.
• 4. Run the basic lab with docker compose run --rm eval-runner. 5. Enable fault injection with docker compose --profile fault run --rm eval-runner. 6. Record an environment manifest JSON describing caseId, datasetRevision, promptRevision, policyRevision, toolSchemaRevision, model, temperature, seed, and clock. 7. Route tool calls to the fixture‑api using environment variables and case‑specific headers. 8. Persist evidence (environment.json, results.json, junit.xml, traces/, README.md) to a unique ./evidence/run‑<timestamp> directory. 9. Create a machine‑readable comparison.json when evaluating against a baseline. 10. Upload the evidence directory to CI even on failure, ensuring a COMPLETE marker after all artifacts are flushed.

证据

Containers cannot make an external model deterministic. They can remove a large amount of accidental variability around it.

来源

Persist evidence outside the containers. Do not let the only result be a process exit code.

来源