Freeze a Real Run into a Corpus: How Baize Scores Agents with Deterministic Evidence

说明与用途

Baize 通过从真实运行事件流重建确定性工具调用时间线，并根据显式断言进行评分，来检测代理的隐藏工具调用错误，而不使用 LLM 作为评判者。

问题

Evaluating agent behavior by inspecting streaming replies misses tool‑call errors such as approving unread tickets, calling forbidden write endpoints, or executing actions out of order.

结果

By scoring deterministic tool‑call timelines against explicit assertions, Baize provides reproducible, LLM‑free evaluation of agent behavior, enabling teams to detect silent tool‑call errors and maintain a human‑agreement‑based calibration loop.

工作流

• 1. Record real runs and export their event streams.
• 2. Use the settings page to sample historical runs by filters and deterministically select a batch.
• 3. Ingest each run into an EvalCase, freezing its event stream and input snapshot.
• 4. Fill in expectations (required_reads, consequential, forbid_tools, etc.) in the UI to confirm the case.
• 5. Run the deterministic scorer to rebuild the tool‑call timeline and apply assertions, yielding failure codes.
• 6. Optionally perform a live state check against downstream snapshots.
• 7. Export confirmed cases to JSON for offline replay or CI.
• 8. Use the human calibration layer to compare automatic verdicts with human agree/disagree annotations.
• 9. Iterate: tighten or relax rules and monitor agreement rate.
• 10. Convert problematic trajectories into skill drafts for reuse.

证据

Baize evaluates agents on the tool‑call timeline instead: not what the model said, but what it actually did — and by default the scoring is deterministic code rules, not an LLM judge.

来源

The same events plus the same assertions always produce the same score: no judge-model noise, no scoring tokens consumed, safe to replay inside any automation or CI.

来源

For each already-scored confirmed case, a human can mark "agree / disagree" and leave a note; the system aggregates an agreement rate (fraction where the automatic verdict matches the human one).

来源