I let an AI agent write my Playwright API tests. It caught a bug my suite missed

Описание и применение

AI‑генерируемые тесты Playwright обнаружили все 10 внедренных ошибок, включая фильтр поиска, который пропустил ручной набор. Однако они содержали больше ошибок линтера и слабых утверждений, особенно в негативных тестах.

Задача

A hand‑written Playwright test suite missed a search‑filter bug that an AI‑generated suite caught.

Результат

The AI‑generated suites caught all ten injected faults, outperforming the hand‑written suite which missed the search‑filter bug. However, the AI suites produced more lint violations and weaker assertions, especially in negative tests, indicating that rule files improved but did not eliminate shallow assertions.

Рабочий процесс

• 1. Build a local stub of the API with ten fault switches.
• 2. Write a hand‑written test suite (H) covering seven behaviours.
• 3. Run H against the clean stub and each fault to record caught faults.
• 4. Prompt Claude Code 2.1.304 to generate suite A with no rules.
• 5. Prompt the same agent to generate suite B with a 7‑rule file (no hard waits, assert on API return, validate schema, clean data, cover negative paths, run linter, write bug comment).
• 6. Run each suite against the clean stub and each fault, recording caught faults, pass rates, flakiness, lint violations, type errors, and weak assertions.
• 7. Compare results.
• 8. Analyse why suite A missed fault 6 (negative search test missing).
• 9. Analyse lint and type errors in AI suites.
• 10. Summarise findings.

Доказательства

The agent’s suites caught more bugs than mine.

Источник

Suite A, with no rules at all, added a second search with a last name that cannot exist... fault 6 is caught.

Источник

Suite A had 17 errors, all from the same family: no-unsafe-assignment, no-unsafe-member-access, no-unsafe-return.

Источник