I let an AI agent write my Playwright API tests. It caught a bug my suite missed

Description and use

AI‑generated Playwright tests caught all 10 injected faults, including a search‑filter bug missed by the hand‑written suite. They, however, produced more lint violations and weaker assertions, especially in negative tests.

Problem

A hand‑written Playwright test suite missed a search‑filter bug that an AI‑generated suite caught.

Outcome

The AI‑generated suites caught all ten injected faults, outperforming the hand‑written suite which missed the search‑filter bug. However, the AI suites produced more lint violations and weaker assertions, especially in negative tests, indicating that rule files improved but did not eliminate shallow assertions.

Workflow

• 1. Build a local stub of the API with ten fault switches.
• 2. Write a hand‑written test suite (H) covering seven behaviours.
• 3. Run H against the clean stub and each fault to record caught faults.
• 4. Prompt Claude Code 2.1.304 to generate suite A with no rules.
• 5. Prompt the same agent to generate suite B with a 7‑rule file (no hard waits, assert on API return, validate schema, clean data, cover negative paths, run linter, write bug comment).
• 6. Run each suite against the clean stub and each fault, recording caught faults, pass rates, flakiness, lint violations, type errors, and weak assertions.
• 7. Compare results.
• 8. Analyse why suite A missed fault 6 (negative search test missing).
• 9. Analyse lint and type errors in AI suites.
• 10. Summarise findings.

Evidence

The agent’s suites caught more bugs than mine.

Source

Suite A, with no rules at all, added a second search with a last name that cannot exist... fault 6 is caught.

Source

Suite A had 17 errors, all from the same family: no-unsafe-assignment, no-unsafe-member-access, no-unsafe-return.

Source