I built Agent Review Studio: a local-first workbench for agent harness evaluations

Description and use

Engineers now have a tool to deeply analyze AI agent runs, view claims, sources, and execution traces, label failures, and refine models without automatic fine-tuning.

Problem

Engineers lacked a comprehensive way to inspect AI agent runs beyond final answers, making it hard to trace claims, evidence, and execution details.

Outcome

Engineers can now systematically review agent runs, identify failures, generate golden evaluation cases, and refine prompts, tools, and memory policies, improving agent quality without automatic fine-tuning.

Workflow

• Create a workspace for the agent or harness
• Run or import a versioned evaluation
• Inspect claims alongside source evidence and execution traces
• Label failures and write corrections
• Score the run across five quality dimensions
• Preserve immutable review history
• Compare baseline and candidate runs
• Export reviewed improvement data

Evidence

Agent Review Studio lets an engineer: Create a workspace for any agent or harness. Run or import a versioned evaluation. Inspect claims beside their source evidence and execution traces. Label failures and write the correction. Score the complete run across five quality dimensions. Preserve immutable review history. Compare baseline and candidate runs. Export reviewed improvement data.

Source