The Agent Loop Was Fast. The Trace Lied.

Описание и применение

Разделение измерения времени на три отдельные зоны (подготовка запроса, вызов модели, выполнение инструмента) выявило, что основная задержка была в TTFT и выполнении инструментов, а не в модели.

Задача

A single timer wrapped the entire agent loop, masking the true source of latency and leading to incorrect optimization decisions.

Результат

Separating the timing into three spans exposed that TTFT and tool execution, not the model, were the primary latency sources, enabling targeted optimizations.

Рабочий процесс

• 1. Wrap each logical phase in a context manager that records start and end times.
• 2. In the prompt_build span, log payload size and other metadata.
• 3. In the model_round_trip span, stream the response, capture TTFT on the first content chunk, and log token usage.
• 4. In the tool_exec span, execute the tool and record success status.
• 5. Append each span as a JSONL line to a trace file for post‑hoc analysis.

Доказательства

You wrap the entire round trip in one clock. The number comes back huge, so you blame the model and start shopping for a smaller one.

Источник

You need three spans, not one timer.

Источник