I let a one-shot Sonnet 5 agent QA a live app. Then I ran our QA tool on the same one

Description and use

A one‑shot Sonnet 5 agent found some bugs but produced far fewer tests than the company’s automated QA pipeline, which also caught bugs the agent missed. The experiment shows that a single LLM run is insufficient for comprehensive QA.

Problem

Can a single run of a large language model (LLM) agent replace a full QA system for production applications?

Outcome

The automated QA pipeline produced a far larger, more comprehensive, and repeatable suite of tests, and it caught bugs that the one‑shot agent missed. The agent was useful for a quick exploratory run but insufficient as a standalone QA solution.

Workflow

• 1. Deploy the production EU rail‑booking app for testing.
• 2. Configure a one‑shot Sonnet 5 agent with no hand‑holding to explore the app and generate tests and bug reports.
• 3. Run the company’s automated QA pipeline on the same app to generate executable test cases and run them.
• 4. Collect outputs from both the agent and the pipeline, anonymize them, and present them to an independent judge.
• 5. Judge evaluates feature coverage, test‑case depth, edge cases, and bug detection.
• 6. Analyze results and identify gaps between the two approaches.
• 7. Incorporate findings into future QA improvements.

Evidence

Our tool wrote 415 executable, grounded test cases. The one‑shot agent wrote 36.

Source

Then the part I didn’t expect: our tool caught bugs the agent walked straight past.

Source