Freeze a Real Run into a Corpus: How Baize Scores Agents with Deterministic Evidence

Description and use

Baize evaluates agents by reconstructing deterministic tool‑call timelines from real run event streams, scoring them against explicit assertions to detect silent tool‑call errors without an LLM judge.

Problem

Evaluating agent behavior by inspecting streaming replies misses tool‑call errors such as approving unread tickets, calling forbidden write endpoints, or executing actions out of order.

Outcome

By scoring deterministic tool‑call timelines against explicit assertions, Baize provides reproducible, LLM‑free evaluation of agent behavior, enabling teams to detect silent tool‑call errors and maintain a human‑agreement‑based calibration loop.

Workflow

• 1. Record real runs and export their event streams.
• 2. Use the settings page to sample historical runs by filters and deterministically select a batch.
• 3. Ingest each run into an EvalCase, freezing its event stream and input snapshot.
• 4. Fill in expectations (required_reads, consequential, forbid_tools, etc.) in the UI to confirm the case.
• 5. Run the deterministic scorer to rebuild the tool‑call timeline and apply assertions, yielding failure codes.
• 6. Optionally perform a live state check against downstream snapshots.
• 7. Export confirmed cases to JSON for offline replay or CI.
• 8. Use the human calibration layer to compare automatic verdicts with human agree/disagree annotations.
• 9. Iterate: tighten or relax rules and monitor agreement rate.
• 10. Convert problematic trajectories into skill drafts for reuse.

Evidence

Baize evaluates agents on the tool‑call timeline instead: not what the model said, but what it actually did — and by default the scoring is deterministic code rules, not an LLM judge.

Source

The same events plus the same assertions always produce the same score: no judge-model noise, no scoring tokens consumed, safe to replay inside any automation or CI.

Source

For each already-scored confirmed case, a human can mark "agree / disagree" and leave a note; the system aggregates an agreement rate (fraction where the automatic verdict matches the human one).

Source