Let the AI click. Don’t let it grade.

Description and use

The article discusses integrating AI into e2e tests, proposing a hybrid approach: use agent.act for actions, expect for deterministic checks, and agent.assert only for semantic validations. This reduces model costs, improves reliability, and eases regression detection.

Problem

Determining which test steps should invoke an AI model versus which should remain deterministic code to balance cost, reliability, and maintainability in end‑to‑end testing.

Outcome

Adopting a hybrid AI‑driven testing strategy reduced model calls from every step to only interaction steps, lowered costs, improved determinism, and made regressions easier to detect.

Workflow

• 1. Write tests using agent.act for actions and expect for deterministic checks.
• 2. Commit the .e2e/cache/ folder to version control and enable --strict-cache in CI.
• 3. Run tests; agent.act steps replay from cache unless stale, while expect assertions validate UI state.
• 4. Use agent.assert only for outcomes that cannot be expressed as locators, such as chatbot responses.

Evidence

"agent.act" hands a goal in plain English to a model that drives the app.

Source

"agent.assert" always run live. A judgment is never cached.

Source

The recording is a 1.4 KB JSON file. It holds two actions…

Source

The product broke. My dead button: exit 1, ASSERTION_FAILED…

Source