Your AI Agent Passed Every Answer Test — and Still Broke Production

Описание и применение

Проверка AI‑агента должна учитывать не только правильность финального ответа, но и путь выполнения, включая вызовы инструментов и соблюдение политик.

Задача

An AI agent can produce the correct final answer while performing unauthorized actions, such as writing to a CRM before human approval, which is not detected by traditional answer‑only evaluations.

Результат

The evaluation harness successfully flagged a run where the agent performed a CRM update without approved authorization, demonstrating that final‑answer checks alone are insufficient.

Рабочий процесс

• Capture structured events with tool call details and context
• Define non‑negotiable invariants (e.g., approval required for writes, scope limits, budget limits)
• Implement a Python evaluation function that checks final answer, cost, and event compliance
• Run the evaluator against agent runs to identify failures
• Add adversarial scenarios to test edge cases (timeouts, denied approvals, cross‑customer access)
• Score outcome and behavior separately, reporting granular pass/fail per dimension
• Validate external effects by checking target systems (CRM records, emails) to confirm actual state changes

Доказательства

A production agent can return the right text after making the wrong API call, skipping approval, or writing to a system it should have left untouched.

Источник

The event fields above are a simplified illustration, not a universal agent trace schema.

Источник

Output: {'passed': False, 'failures': ['Write performed without approved authorization']}

Источник