Build an Agent Test Harness: Deterministic State Assertions in CI

Описание и применение

Создайте тестовый harness для AI‑агентов, проверяющий реальные изменения в системе, а не только текстовый вывод, и интегрируйте его в CI.

Задача

AI agents often appear successful by text output while failing to change the underlying system state, leading to unreliable automation and project cancellations.

Результат

Agents are validated against deterministic state changes, reducing false positives and aligning CI failures with real system corruption.

Рабочий процесс

• 1. Fixture the world – seed test DB, mock HTTP APIs, set controlled clock.
• 2. Drive the agent – submit a fixed goal with bounded steps and tokens.
• 3. Capture traces – record every tool call, argument, response, and plan.
• 4. Assert on state – compare post‑run world to expected ground‑truth snapshot.
• 5. Bound behavior – enforce step limits and allowlist of tools.
• 6. Store trace artifacts for debugging failed builds.

Доказательства

An agent that says "Task completed successfully" can leave the database untouched, call the wrong tool with the right-looking arguments, or finish four of five steps and still break the workflow.

Источник

The harness does four jobs: Fixture the world, Drive the agent, Capture traces, Assert on state.

Источник

Wire the harness so a pull request cannot merge when terminal state fails.

Источник