Your LLM Was Right. Your AI Agent Still Shipped the Wrong Answer

Описание и применение

С помощью Stepfork удалось зафиксировать и воспроизвести работу ИИ‑агента, выявив ошибку бизнес‑логики, которая скрывала правильное решение модели Gemini.

Задача

An AI agent correctly identified a critical incident, but application logic silently overrode the model’s decision, producing an incorrect business outcome.

Результат

Using Stepfork, the team captured the agent’s execution, replayed it offline, and turned the failure into a pytest regression test. The test revealed a business‑logic bug that was fixed without additional LLM calls, demonstrating efficient debugging of AI agents.

Рабочий процесс

• Record agent execution with Stepfork, capturing LLM and tool interactions
• Export the trace to a pytest regression test specifying expected business outcome
• Run the generated test against the buggy application to detect mismatch
• Modify application logic to remove incorrect deployment correlation
• Re‑run the same pytest test to confirm pass without new LLM calls

Доказательства

I built an open‑source Python tool to record AI agent runs, replay failures offline, and turn them into pytest regression tests.

Источник

The model returned: {"severity":"P0"}

Источник

The final output became: {"severity":"P1","escalation_required":false}

Источник

stepfork export traces/incident-triage.sftrace \
--pytest \
--entrypoint agent:run_incident_agent \
--expect-output expected.json \
--output tests/test_incident_regression.py

Источник