I let a one-shot Sonnet 5 agent QA a live app. Then I ran our QA tool on the same one

Описание и применение

Один раз запущенный агент Sonnet 5 обнаружил несколько ошибок, но его набор тестов был значительно меньше, чем у автоматизированной системы QA, которая выявила больше багов и создала более обширный набор воспроизводимых тестов.

Задача

Can a single run of a large language model (LLM) agent replace a full QA system for production applications?

Результат

The automated QA pipeline produced a far larger, more comprehensive, and repeatable suite of tests, and it caught bugs that the one‑shot agent missed. The agent was useful for a quick exploratory run but insufficient as a standalone QA solution.

Рабочий процесс

• 1. Deploy the production EU rail‑booking app for testing.
• 2. Configure a one‑shot Sonnet 5 agent with no hand‑holding to explore the app and generate tests and bug reports.
• 3. Run the company’s automated QA pipeline on the same app to generate executable test cases and run them.
• 4. Collect outputs from both the agent and the pipeline, anonymize them, and present them to an independent judge.
• 5. Judge evaluates feature coverage, test‑case depth, edge cases, and bug detection.
• 6. Analyze results and identify gaps between the two approaches.
• 7. Incorporate findings into future QA improvements.

Доказательства

Our tool wrote 415 executable, grounded test cases. The one‑shot agent wrote 36.

Источник

Then the part I didn’t expect: our tool caught bugs the agent walked straight past.

Источник