The Results Were SCARY: One Brief, 20 AI-Built Payment Apps

Description and use

Twenty independent AI agents built 20 distinct payment apps, showing varied behavior and final balances, highlighting the need for control and verification.

Problem

Can 20 independent AI agents produce identical behavior from the same requirements?

Outcome

The 20 AI-built payment apps behaved differently: 2xx responses ranged from 244 to 260 per implementation, and final account balances varied significantly. The experiment shows that identical requirements do not guarantee identical behavior when agents work in isolation.

Workflow

• Write shared requirements and AGENTS.md instructions
• Orchestrate 20 folders and assign one agent per folder
• Each agent builds independently without cross‑feedback or human review
• Run a fixed fixture of 10 accounts and 500 payment messages per implementation
• Record HTTP status, payment status, response body, elapsed time, expected case, and timestamp
• Compare source code, static analysis signals, and final account balances

Evidence

The 20 applications did not behave identically on the same stateful sequence.

Source

Their 2xx response totals ranged from 244 of 500 for implementation 17 to 260 of 500 for implementation 08.

Source

For example, ACC-1001 started at 500.000 JOD and ended between 3.576 and 4.816 JOD across the runs.

Source