Playwright Tests Need an Evidence Ledger

Description and use

Implementing an evidence ledger in Playwright tests enables precise correlation of email requests and responses, eliminating ambiguous timing failures and improving CI reliability.

Problem

Email verification tests often fail due to timing issues, but the failures hide the real problem: tests lack a record of which message they expected and why it was selected.

Outcome

Implementing an evidence ledger transforms ambiguous timing failures into actionable diagnostics, reduces flaky tests, and improves maintainability of email verification workflows.

Workflow

• Assign a unique run ID before creating an account.
• Record the input email, request time, polling selector, and message ID returned by the inbox adapter.
• Use a correlation key (header or plus‑address) to match the message to the request.
• Poll with a deadline, interval, and typed failure reasons (e.g., message_not_found, provider_unavailable, invalid_correlation).
• Attach the in‑memory evidence array to the test report when the test ends.
• Print a compact ledger summary next to the Playwright trace on failure.
• Ensure cleanup records success or explains failure to avoid contaminating subsequent tests.

Evidence

Email verification tests often get labeled as timing problems. The test waits for a message, the message arrives, and the assertion still fails.

Source

An evidence ledger gives each Playwright run a small, inspectable history. It connects the signup action, the email request, the message that was observed, and the final assertion.

Source

If message-observed has no email-requested event, the test used stale evidence.

Source