Your AI Agent Passed Every Answer Test — and Still Broke Production

Description and use

Evaluating AI agents requires inspecting both final answers and execution traces to detect unauthorized actions, such as premature CRM writes.

Problem

An AI agent can produce the correct final answer while performing unauthorized actions, such as writing to a CRM before human approval, which is not detected by traditional answer‑only evaluations.

Outcome

The evaluation harness successfully flagged a run where the agent performed a CRM update without approved authorization, demonstrating that final‑answer checks alone are insufficient.

Workflow

• Capture structured events with tool call details and context
• Define non‑negotiable invariants (e.g., approval required for writes, scope limits, budget limits)
• Implement a Python evaluation function that checks final answer, cost, and event compliance
• Run the evaluator against agent runs to identify failures
• Add adversarial scenarios to test edge cases (timeouts, denied approvals, cross‑customer access)
• Score outcome and behavior separately, reporting granular pass/fail per dimension
• Validate external effects by checking target systems (CRM records, emails) to confirm actual state changes

Evidence

A production agent can return the right text after making the wrong API call, skipping approval, or writing to a system it should have left untouched.

Source

The event fields above are a simplified illustration, not a universal agent trace schema.

Source

Output: {'passed': False, 'failures': ['Write performed without approved authorization']}

Source