Cypress Email Tests Need a Failure Ledger

Description and use

Implementing a failure ledger in Cypress provides detailed records of each email verification attempt, improving diagnosis and reducing flaky retries.

Problem

Email verification tests in Cypress often fail with vague messages, making it hard to diagnose failures and leading to flaky retries.

Outcome

Implementing a failure ledger reduces flaky retries, provides clear failure explanations, and improves QA triage efficiency.

Workflow

• Allocate a unique mailbox or lease before test action.
• Save trigger timestamp immediately before user action.
• Poll mailbox until a bounded deadline, logging candidate message IDs without secrets.
• Reject messages older than trigger or from other runs.
• Accept a message only when recipient and correlation rules match.
• Write a compact ledger entry (test name, CI job, browser, worker, retry, mailbox id, trigger time, filters, message ids, acceptance/rejection reasons, assertion error, artifact path).
• Release mailbox after failure.
• Include machine/worker ID when running in parallel.
• Use the ledger to triage failures before adding retries.

Evidence

A failure ledger gives each attempt a short, structured story. It records what the test requested, what the mailbox returned, and why the assertion accepted or rejected a message.

Source

Keep the ledger compact and redact secrets. A useful entry usually contains: test name, CI job, browser, worker, and retry number…

Source

Use a bounded deadline rather than an open-ended wait. A very large timeout turns a provider outage into a slow CI outage.

Source