Find where two AI agent runs diverged

说明与用途

使用 NovaFabric 逐步识别两次 AI 代理运行差异的流程：捕获、比较和 CI 集成。

问题

Engineers need to identify why two identical agent runs produce different outputs.

结果

The workflow demonstrates how to pinpoint divergence in agent runs by capturing run capsules, diffing them, and interpreting the diff to locate model response changes.

工作流

• Start mock LLM server with environment variables
• Capture first run with nova capture and mode bad
• Capture second run with nova capture and mode fixed
• Run nova diff on the two capsule paths
• Interpret diff output: aligned calls, request_changed, response_changed, outputs changed
• Use nova diff --assert-no-regressions as a CI gate
• Configure GitHub Actions to output annotations

证据

NovaFabric is open source (Apache-2.0) and pre-1.0: https://github.com/MSKazemi/novafabric

来源

The diff reports request_changed: false, response_changed: true

来源

Use nova diff --assert-no-regressions as a CI gate

来源