What agent traces can tell you without an LLM judge

Description and use

tracelint detects agent defects from traces alone, flagging failures like deploying an unstable build.

Problem

Agent failures can occur even when a run reports success, making it hard to detect defects without manual inspection.

Outcome

tracelint detected a hard defect: a failed pipeline (UNSTABLE) was deployed, causing CI to fail with exit code 2.

Workflow

• Collect OpenInference spans from Phoenix
• Create tools.json with failure_when and side_effecting metadata
• Run tracelint check on spans.json
• Review findings: hard defects, candidates, not checked
• Adjust tools.json or agent logic to prevent future defects

Evidence

A useful rule for deciding which is which: if the verdict depends on what the agent should have meant, it's an eval. If it depends only on what the trace records, it can be a deterministic check.

Source

Run tracelint on this trace as-is and it won't fail it, and that's correct: nothing in the trace says UNSTABLE means failure.

Source

The pipeline failed. Its result was UNSTABLE, which your tools.json says counts as a failure.

Source