The Results Were SCARY: One Brief, 20 AI-Built Payment Apps

说明与用途

20个独立的AI代理构建了20个不同的支付应用，表现出行为和最终余额的差异，强调了控制和验证的重要性。

问题

Can 20 independent AI agents produce identical behavior from the same requirements?

结果

The 20 AI-built payment apps behaved differently: 2xx responses ranged from 244 to 260 per implementation, and final account balances varied significantly. The experiment shows that identical requirements do not guarantee identical behavior when agents work in isolation.

工作流

• Write shared requirements and AGENTS.md instructions
• Orchestrate 20 folders and assign one agent per folder
• Each agent builds independently without cross‑feedback or human review
• Run a fixed fixture of 10 accounts and 500 payment messages per implementation
• Record HTTP status, payment status, response body, elapsed time, expected case, and timestamp
• Compare source code, static analysis signals, and final account balances

证据

The 20 applications did not behave identically on the same stateful sequence.

来源

Their 2xx response totals ranged from 244 of 500 for implementation 17 to 260 of 500 for implementation 08.

来源

For example, ACC-1001 started at 500.000 JOD and ended between 3.576 and 4.816 JOD across the runs.

来源