I Gave the Same Governance Audit to One AI and an AI Swarm. They Missed the Same Bug.

Description and use

The experiment showed that AI orchestration improves coverage but does not eliminate attention issues and can lose specific evidence.

Problem

Determining whether a single AI or an orchestrated swarm can reliably detect governance defects across a large corpus, and understanding the trade-offs between coverage, attention, and evidence preservation.

Outcome

The swarm achieved full document assignment and richer cross‑domain findings but still missed a mechanical reference bug and lost specificity in evidence, revealing that orchestration improves coverage but can dilute actionable evidence.

Workflow

• 1. Prepare a sealed benchmark of eight conditions within the governance corpus.
• 2. Mount the corpus read‑only in a Docker container.
• 3. Run Configuration A: prompt a single Claude Opus 5 to audit the corpus without prior context or orchestration.
• 4. Record findings and runtime (≈14 min).
• 5. Run Configuration B: coordinator partitions all 341 documents, assigns them to three analyst agents, and a synthesis agent consolidates reports.
• 6. Record findings and runtime (≈43 min).
• 7. Compare results against the hidden benchmark.
• 8. Analyze coverage, attention, and evidence preservation differences.
• 9. Document lessons on deterministic rule enforcement and synthesis impact.

Evidence

The swarm did things the single agent could not.

Source

Both missed the broken reference.

Source