Find where two AI agent runs diverged

Description and use

Step‑by‑step process to identify differences in two AI agent runs using NovaFabric: capture, diff, and CI integration.

Problem

Engineers need to identify why two identical agent runs produce different outputs.

Outcome

The workflow demonstrates how to pinpoint divergence in agent runs by capturing run capsules, diffing them, and interpreting the diff to locate model response changes.

Workflow

• Start mock LLM server with environment variables
• Capture first run with nova capture and mode bad
• Capture second run with nova capture and mode fixed
• Run nova diff on the two capsule paths
• Interpret diff output: aligned calls, request_changed, response_changed, outputs changed
• Use nova diff --assert-no-regressions as a CI gate
• Configure GitHub Actions to output annotations

Evidence

NovaFabric is open source (Apache-2.0) and pre-1.0: https://github.com/MSKazemi/novafabric

Source

The diff reports request_changed: false, response_changed: true

Source

Use nova diff --assert-no-regressions as a CI gate

Source