I gave AI agents a harm budget instead of trust and ran 19 experiments

Описание и применение

Разработан подход к оценке доверия к ИИ‑агентам через бюджет вреда, что позволило снизить финансовые потери и повысить контроль над автономией.

Задача

Developers and customers lacked a reliable way to assess whether an AI agent could be trusted to act autonomously without causing financial loss.

Результат

Agents with orchestrators averaged $470 profit per run versus $5,750 loss without; harm budget enforcement reduced total harm and improved trust.

Рабочий процесс

• 1. Design agent passport and orchestrator to monitor actions
• 2. Define hidden exam tasks with known correct answers and associated monetary harm per mistake
• 3. Set a harm budget per period and calculate allowed autonomous actions
• 4. Run sandbox experiments with fake shop, customers, and money
• 5. Compare agent performance with and without orchestrator
• 6. Analyze results, adjust orchestrator behavior, and refine harm calculations

Доказательства

I care how much money each action the agent takes on its own loses on average.

Источник

Agents that learned from someone else's checked experience reached full autonomy by round 5, agents learning alone took until round 7 or 9.

Источник

The model costs about $0.002 per ticket. Paid checks, human time and mostly harm make up more than 99 percent of the cost.

Источник