Trace First, Blame Later: A Debug Loop for Failing Agent Runs

Description and use

A step‑by‑step workflow to capture logs, tool calls, and diffs for LLM agent debugging, enabling quick failure isolation.

Problem

Agent runs often finish with exit code 0 yet produce incorrect side‑effects, making it hard to determine whether the model or the harness failed.

Outcome

The workflow provides a clear, repeatable process to capture all signals of an agent run, enabling teams to pinpoint failures and reduce debugging effort.

Workflow

• 1. Ensure a git repo, tool‑call stream, and repeatable command exist.
• 2. Run the agent with the trace_bundle.sh script to produce a timestamped bundle in traces/.
• 3. Review the bundle: before.patch, after.patch, agent.log, and tool_calls.jsonl.
• 4. Summarize the tool stream with the provided Python script to identify suspicious calls.
• 5. Apply the debug loop: reproduce, reduce, isolate, fix, repeating until the bundle is clean.

Evidence

The script below wraps any agent command. It produces a timestamped bundle in traces/.

Source

Four steps. Repeat until the bundle looks clean.

Source