Build an Agent Test Harness: Deterministic State Assertions in CI

Description and use

Build a deterministic state assertion harness for AI agents, ensuring CI fails when real system changes are missing.

Problem

AI agents often appear successful by text output while failing to change the underlying system state, leading to unreliable automation and project cancellations.

Outcome

Agents are validated against deterministic state changes, reducing false positives and aligning CI failures with real system corruption.

Workflow

• 1. Fixture the world – seed test DB, mock HTTP APIs, set controlled clock.
• 2. Drive the agent – submit a fixed goal with bounded steps and tokens.
• 3. Capture traces – record every tool call, argument, response, and plan.
• 4. Assert on state – compare post‑run world to expected ground‑truth snapshot.
• 5. Bound behavior – enforce step limits and allowlist of tools.
• 6. Store trace artifacts for debugging failed builds.

Evidence

An agent that says "Task completed successfully" can leave the database untouched, call the wrong tool with the right-looking arguments, or finish four of five steps and still break the workflow.

Source

The harness does four jobs: Fixture the world, Drive the agent, Capture traces, Assert on state.

Source

Wire the harness so a pull request cannot merge when terminal state fails.

Source