Email Checks Need a Failure Budget

Description and use

Email verification tests often fail due to vague timeouts and retries. By defining a failure budget, recording timestamps, and saving lightweight receipts, tests become bounded, diagnosable, and faster in CI.

Problem

Email verification tests often fail silently due to unclear timeouts and retries, making debugging slow and CI expensive.

Outcome

Tests become predictable: bounded wait times, clear failure reasons, and efficient CI usage. Debugging shifts from vague timeouts to concrete evidence of queueing, delivery, and retry causes.

Workflow

• 1. Define a budget object with totalMs, pollEveryMs, maxAttempts, and subject.
• 2. Record a startedAt timestamp before triggering the email.
• 3. Poll the inbox at pollEveryMs until totalMs expires, filtering messages after startedAt.
• 4. If a matching subject is found, return match info and poll count.
• 5. On timeout, throw an error with poll count.
• 6. Before retrying, save a receipt containing run ID, inbox ID, trigger timestamp, poll number, elapsed time, matching IDs, and failure reason.
• 7. Retry only if the receipt indicates a transient delivery issue; otherwise fail immediately.
• 8. Log all receipts and poll data for both passes and failures.

Evidence

"If those clocks are mixed together, a timeout says very little. Maybe the app never queued the message. Maybe the message arrived after the final poll. Maybe the test read an older message."

Source

"I also record a startedAt timestamp before the action that triggers the email. Every inbox query filters for messages newer than that boundary."

Source

"Before the next attempt, save a small receipt. It does not need to contain the message body or private user data."

Source