The Agent That Refunded Everyone

Описание и применение

AI‑агент, выдающий возвраты без проверки заказа, был исправлен внедрением цикла с тестированием, трассировкой, оценкой и кодовыми правилами.

Задача

An AI refund agent issued refunds without verifying orders after users exploited a prompt trick.

Результат

The agent was fixed by integrating a systematic loop that caught the flaw before launch, preventing unauthorized refunds.

Рабочий процесс

• Red team the agent with adaptive, multi-turn attacks
• Trace every finding to the exact step that failed
• Turn findings into evals that run on every release
• Enforce policies as guardrails in code and monitor them in production

Доказательства

The agent believed them. It skipped order verification, because the "approval" sounded official, and issued the refund.

Источник

Nothing checked whether verify_order was called before issue_refund.

Источник

Capture prompts, context, tool calls and the execution path for every run. You can't fix what you can't see.

Источник

Rules that must never break belong in code, between the model and the tools.

Источник