Trace First, Blame Later: A Debug Loop for Failing Agent Runs

Описание и применение

Пошаговый процесс сбора логов, вызовов инструментов и diff‑ов для отладки LLM‑агентов, позволяющий быстро выявлять ошибки.

Задача

Agent runs often finish with exit code 0 yet produce incorrect side‑effects, making it hard to determine whether the model or the harness failed.

Результат

The workflow provides a clear, repeatable process to capture all signals of an agent run, enabling teams to pinpoint failures and reduce debugging effort.

Рабочий процесс

• 1. Ensure a git repo, tool‑call stream, and repeatable command exist.
• 2. Run the agent with the trace_bundle.sh script to produce a timestamped bundle in traces/.
• 3. Review the bundle: before.patch, after.patch, agent.log, and tool_calls.jsonl.
• 4. Summarize the tool stream with the provided Python script to identify suspicious calls.
• 5. Apply the debug loop: reproduce, reduce, isolate, fix, repeating until the bundle is clean.

Доказательства

The script below wraps any agent command. It produces a timestamped bundle in traces/.

Источник

Four steps. Repeat until the bundle looks clean.

Источник