I built Agent Review Studio: a local-first workbench for agent harness evaluations

Описание и применение

Инженеры получили инструмент для детального анализа работы AI‑агентов, позволяющий просматривать утверждения, источники и трассировки, отмечать ошибки и улучшать модели без автоматического обучения.

Задача

Engineers lacked a comprehensive way to inspect AI agent runs beyond final answers, making it hard to trace claims, evidence, and execution details.

Результат

Engineers can now systematically review agent runs, identify failures, generate golden evaluation cases, and refine prompts, tools, and memory policies, improving agent quality without automatic fine-tuning.

Рабочий процесс

• Create a workspace for the agent or harness
• Run or import a versioned evaluation
• Inspect claims alongside source evidence and execution traces
• Label failures and write corrections
• Score the run across five quality dimensions
• Preserve immutable review history
• Compare baseline and candidate runs
• Export reviewed improvement data

Доказательства

Agent Review Studio lets an engineer: Create a workspace for any agent or harness. Run or import a versioned evaluation. Inspect claims beside their source evidence and execution traces. Label failures and write the correction. Score the complete run across five quality dimensions. Preserve immutable review history. Compare baseline and candidate runs. Export reviewed improvement data.

Источник