What agent traces can tell you without an LLM judge

说明与用途

tracelint 通过仅使用跟踪信息检测代理错误，避免使用 LLM 判定器。

问题

Agent failures can occur even when a run reports success, making it hard to detect defects without manual inspection.

结果

tracelint detected a hard defect: a failed pipeline (UNSTABLE) was deployed, causing CI to fail with exit code 2.

工作流

• Collect OpenInference spans from Phoenix
• Create tools.json with failure_when and side_effecting metadata
• Run tracelint check on spans.json
• Review findings: hard defects, candidates, not checked
• Adjust tools.json or agent logic to prevent future defects

证据

A useful rule for deciding which is which: if the verdict depends on what the agent should have meant, it's an eval. If it depends only on what the trace records, it can be a deterministic check.

来源

Run tracelint on this trace as-is and it won't fail it, and that's correct: nothing in the trace says UNSTABLE means failure.

来源

The pipeline failed. Its result was UNSTABLE, which your tools.json says counts as a failure.

来源