Your LLM Was Right. Your AI Agent Still Shipped the Wrong Answer

Description and use

Stepfork enabled recording and replay of an AI agent’s execution, turning a business‑logic bug into a pytest regression test that was fixed without new LLM calls.

Problem

An AI agent correctly identified a critical incident, but application logic silently overrode the model’s decision, producing an incorrect business outcome.

Outcome

Using Stepfork, the team captured the agent’s execution, replayed it offline, and turned the failure into a pytest regression test. The test revealed a business‑logic bug that was fixed without additional LLM calls, demonstrating efficient debugging of AI agents.

Workflow

• Record agent execution with Stepfork, capturing LLM and tool interactions
• Export the trace to a pytest regression test specifying expected business outcome
• Run the generated test against the buggy application to detect mismatch
• Modify application logic to remove incorrect deployment correlation
• Re‑run the same pytest test to confirm pass without new LLM calls

Evidence

I built an open‑source Python tool to record AI agent runs, replay failures offline, and turn them into pytest regression tests.

Source

The model returned: {"severity":"P0"}

Source

The final output became: {"severity":"P1","escalation_required":false}

Source

stepfork export traces/incident-triage.sftrace \
--pytest \
--entrypoint agent:run_incident_agent \
--expect-output expected.json \
--output tests/test_incident_regression.py

Source