Your LLM Was Right. Your AI Agent Still Shipped the Wrong Answer

说明与用途

使用 Stepfork 记录并重放 AI 代理的执行，发现并修复了隐藏正确模型决策的业务逻辑错误。

问题

An AI agent correctly identified a critical incident, but application logic silently overrode the model’s decision, producing an incorrect business outcome.

结果

Using Stepfork, the team captured the agent’s execution, replayed it offline, and turned the failure into a pytest regression test. The test revealed a business‑logic bug that was fixed without additional LLM calls, demonstrating efficient debugging of AI agents.

工作流

• Record agent execution with Stepfork, capturing LLM and tool interactions
• Export the trace to a pytest regression test specifying expected business outcome
• Run the generated test against the buggy application to detect mismatch
• Modify application logic to remove incorrect deployment correlation
• Re‑run the same pytest test to confirm pass without new LLM calls

证据

I built an open‑source Python tool to record AI agent runs, replay failures offline, and turn them into pytest regression tests.

来源

The model returned: {"severity":"P0"}

来源

The final output became: {"severity":"P1","escalation_required":false}

来源

stepfork export traces/incident-triage.sftrace \
--pytest \
--entrypoint agent:run_incident_agent \
--expect-output expected.json \
--output tests/test_incident_regression.py

来源