Build a Reproducible AI Agent Evaluation Lab with Docker Compose

Description and use

Build a reproducible AI agent evaluation lab with Docker Compose, using profiles, pinned images, environment manifests, and evidence directories to eliminate flaky CI failures.

Problem

Agent evaluation failures in CI are hard to diagnose because of non‑deterministic external factors such as tool responses, database state, clock, and dependency versions.

Outcome

The lab enables deterministic AI agent evaluation, reduces flaky CI failures, and provides detailed, reproducible evidence for debugging and regression analysis.

Workflow

• 1. Define core services (e.g., fixture‑api, eval‑runner) in docker-compose.yml with image digests and fixed environment variables.
• 2. Use Compose profiles to separate optional services (fault‑proxy, otel‑collector) that can be enabled with --profile.
• 3. Pin all inputs: container images, JS dependencies, synthetic datasets, prompts, policies, tool schemas, locale, timezone, clock, seed, and provider identifiers.
• 4. Run the basic lab with docker compose run --rm eval-runner. 5. Enable fault injection with docker compose --profile fault run --rm eval-runner. 6. Record an environment manifest JSON describing caseId, datasetRevision, promptRevision, policyRevision, toolSchemaRevision, model, temperature, seed, and clock. 7. Route tool calls to the fixture‑api using environment variables and case‑specific headers. 8. Persist evidence (environment.json, results.json, junit.xml, traces/, README.md) to a unique ./evidence/run‑<timestamp> directory. 9. Create a machine‑readable comparison.json when evaluating against a baseline. 10. Upload the evidence directory to CI even on failure, ensuring a COMPLETE marker after all artifacts are flushed.

Evidence

Containers cannot make an external model deterministic. They can remove a large amount of accidental variability around it.

Source

Persist evidence outside the containers. Do not let the only result be a process exit code.

Source