What agent traces can tell you without an LLM judge

Описание и применение

tracelint позволяет обнаруживать ошибки агента, используя только трассировку, без LLM‑жюри, проверяя, например, что неудачный билд не развернут.

Задача

Agent failures can occur even when a run reports success, making it hard to detect defects without manual inspection.

Результат

tracelint detected a hard defect: a failed pipeline (UNSTABLE) was deployed, causing CI to fail with exit code 2.

Рабочий процесс

• Collect OpenInference spans from Phoenix
• Create tools.json with failure_when and side_effecting metadata
• Run tracelint check on spans.json
• Review findings: hard defects, candidates, not checked
• Adjust tools.json or agent logic to prevent future defects

Доказательства

A useful rule for deciding which is which: if the verdict depends on what the agent should have meant, it's an eval. If it depends only on what the trace records, it can be a deterministic check.

Источник

Run tracelint on this trace as-is and it won't fail it, and that's correct: nothing in the trace says UNSTABLE means failure.

Источник

The pipeline failed. Its result was UNSTABLE, which your tools.json says counts as a failure.

Источник