The Results Were SCARY: One Brief, 20 AI-Built Payment Apps

Описание и применение

20 независимых AI‑агентов создали 20 разных платежных приложений, демонстрируя различия в поведении и финальных балансах, что подчёркивает необходимость контроля и проверки.

Задача

Can 20 independent AI agents produce identical behavior from the same requirements?

Результат

The 20 AI-built payment apps behaved differently: 2xx responses ranged from 244 to 260 per implementation, and final account balances varied significantly. The experiment shows that identical requirements do not guarantee identical behavior when agents work in isolation.

Рабочий процесс

• Write shared requirements and AGENTS.md instructions
• Orchestrate 20 folders and assign one agent per folder
• Each agent builds independently without cross‑feedback or human review
• Run a fixed fixture of 10 accounts and 500 payment messages per implementation
• Record HTTP status, payment status, response body, elapsed time, expected case, and timestamp
• Compare source code, static analysis signals, and final account balances

Доказательства

The 20 applications did not behave identically on the same stateful sequence.

Источник

Their 2xx response totals ranged from 244 of 500 for implementation 17 to 260 of 500 for implementation 08.

Источник

For example, ACC-1001 started at 500.000 JOD and ended between 3.576 and 4.816 JOD across the runs.

Источник