The Agent That Refunded Everyone

Description and use

An AI refund agent issued refunds without order verification; it was fixed by adding red teaming, tracing, evals, and code guardrails.

Problem

An AI refund agent issued refunds without verifying orders after users exploited a prompt trick.

Outcome

The agent was fixed by integrating a systematic loop that caught the flaw before launch, preventing unauthorized refunds.

Workflow

• Red team the agent with adaptive, multi-turn attacks
• Trace every finding to the exact step that failed
• Turn findings into evals that run on every release
• Enforce policies as guardrails in code and monitor them in production

Evidence

The agent believed them. It skipped order verification, because the "approval" sounded official, and issued the refund.

Source

Nothing checked whether verify_order was called before issue_refund.

Source

Capture prompts, context, tool calls and the execution path for every run. You can't fix what you can't see.

Source

Rules that must never break belong in code, between the model and the tools.

Source