Find where two AI agent runs diverged

Описание и применение

Пошаговый процесс выявления различий в двух запусках AI‑агента с помощью NovaFabric: захват, сравнение и интеграция в CI.

Задача

Engineers need to identify why two identical agent runs produce different outputs.

Результат

The workflow demonstrates how to pinpoint divergence in agent runs by capturing run capsules, diffing them, and interpreting the diff to locate model response changes.

Рабочий процесс

• Start mock LLM server with environment variables
• Capture first run with nova capture and mode bad
• Capture second run with nova capture and mode fixed
• Run nova diff on the two capsule paths
• Interpret diff output: aligned calls, request_changed, response_changed, outputs changed
• Use nova diff --assert-no-regressions as a CI gate
• Configure GitHub Actions to output annotations

Доказательства

NovaFabric is open source (Apache-2.0) and pre-1.0: https://github.com/MSKazemi/novafabric

Источник

The diff reports request_changed: false, response_changed: true

Источник

Use nova diff --assert-no-regressions as a CI gate

Источник