I let a one-shot Sonnet 5 agent QA a live app. Then I ran our QA tool on the same one

说明与用途

一次性运行的 Sonnet 5 代理发现了一些错误，但其测试用例数量远少于公司的自动化 QA 流水线，后者还捕获了代理遗漏的错误。实验表明，单次 LLM 运行不足以完成全面 QA。

问题

Can a single run of a large language model (LLM) agent replace a full QA system for production applications?

结果

The automated QA pipeline produced a far larger, more comprehensive, and repeatable suite of tests, and it caught bugs that the one‑shot agent missed. The agent was useful for a quick exploratory run but insufficient as a standalone QA solution.

工作流

• 1. Deploy the production EU rail‑booking app for testing.
• 2. Configure a one‑shot Sonnet 5 agent with no hand‑holding to explore the app and generate tests and bug reports.
• 3. Run the company’s automated QA pipeline on the same app to generate executable test cases and run them.
• 4. Collect outputs from both the agent and the pipeline, anonymize them, and present them to an independent judge.
• 5. Judge evaluates feature coverage, test‑case depth, edge cases, and bug detection.
• 6. Analyze results and identify gaps between the two approaches.
• 7. Incorporate findings into future QA improvements.

证据

Our tool wrote 415 executable, grounded test cases. The one‑shot agent wrote 36.

来源

Then the part I didn’t expect: our tool caught bugs the agent walked straight past.

来源