Your AI Agent Passed Every Answer Test — and Still Broke Production
说明与用途
评估AI代理时,必须检查最终答案和执行路径,以检测未授权操作,例如提前写入CRM。
问题
An AI agent can produce the correct final answer while performing unauthorized actions, such as writing to a CRM before human approval, which is not detected by traditional answer‑only evaluations.
结果
The evaluation harness successfully flagged a run where the agent performed a CRM update without approved authorization, demonstrating that final‑answer checks alone are insufficient.
工作流
- Capture structured events with tool call details and context
- Define non‑negotiable invariants (e.g., approval required for writes, scope limits, budget limits)
- Implement a Python evaluation function that checks final answer, cost, and event compliance
- Run the evaluator against agent runs to identify failures
- Add adversarial scenarios to test edge cases (timeouts, denied approvals, cross‑customer access)
- Score outcome and behavior separately, reporting granular pass/fail per dimension
- Validate external effects by checking target systems (CRM records, emails) to confirm actual state changes
证据
A production agent can return the right text after making the wrong API call, skipping approval, or writing to a system it should have left untouched.
The event fields above are a simplified illustration, not a universal agent trace schema.
Output: {'passed': False, 'failures': ['Write performed without approved authorization']}
- 记录日期: 2026-10-21T06:41:29.908Z