I let an AI agent write my Playwright API tests. It caught a bug my suite missed

说明与用途

AI 生成的 Playwright 测试捕获了所有 10 个注入的错误，包括手写套件遗漏的搜索过滤错误。然而，它们产生了更多的 lint 错误和较弱的断言，尤其是在负面测试中。

问题

A hand‑written Playwright test suite missed a search‑filter bug that an AI‑generated suite caught.

结果

The AI‑generated suites caught all ten injected faults, outperforming the hand‑written suite which missed the search‑filter bug. However, the AI suites produced more lint violations and weaker assertions, especially in negative tests, indicating that rule files improved but did not eliminate shallow assertions.

工作流

• 1. Build a local stub of the API with ten fault switches.
• 2. Write a hand‑written test suite (H) covering seven behaviours.
• 3. Run H against the clean stub and each fault to record caught faults.
• 4. Prompt Claude Code 2.1.304 to generate suite A with no rules.
• 5. Prompt the same agent to generate suite B with a 7‑rule file (no hard waits, assert on API return, validate schema, clean data, cover negative paths, run linter, write bug comment).
• 6. Run each suite against the clean stub and each fault, recording caught faults, pass rates, flakiness, lint violations, type errors, and weak assertions.
• 7. Compare results.
• 8. Analyse why suite A missed fault 6 (negative search test missing).
• 9. Analyse lint and type errors in AI suites.
• 10. Summarise findings.

证据

The agent’s suites caught more bugs than mine.

来源

Suite A, with no rules at all, added a second search with a last name that cannot exist... fault 6 is caught.

来源

Suite A had 17 errors, all from the same family: no-unsafe-assignment, no-unsafe-member-access, no-unsafe-return.

来源