CI Email Tests Need a Run Manifest

Description and use

Adding a run manifest to email tests in CI transforms vague timeouts into detailed, actionable logs, speeding debugging and preventing test pollution.

Problem

Email assertions in CI pipelines lack actionable context, making flaky tests hard to debug.

Outcome

Adding a run manifest to email tests transforms vague timeouts into detailed, actionable logs, reducing debugging time and preventing test pollution.

Workflow

• Create a unique run ID before triggering the application action.
• Record the test name, inbox address, and shard in the manifest.
• Set trigger time, poll deadline, and cleanup deadline timestamps.
• Append seen message IDs and the accepted message ID as they are processed.
• Close the manifest in a finally block, marking status as passed, failed, or expired.
• Save the manifest as an artifact and print a concise summary line on failure.
• Use the manifest to investigate delivery, routing, subject, sender, or link extraction issues.
• Implement an idempotent sweeper to clean up inboxes after expiry or job termination.
• Log decisions such as timestamp filtering or inbox removal in the manifest notes.

Evidence

Email assertions are often the least explainable part of an otherwise tidy CI pipeline.

Source

I have started treating each email test like a small deployment. It gets an identity, a start time, a cleanup deadline, and a receipt.

Source

A useful manifest does not need to become a new database. Four groups of fields cover most debugging sessions: Ownership, Timing, Matching, Lifecycle.

Source

When a test fails, print a one-line summary and save the full manifest as an artifact: email-run signup-1842-attempt-1 failed: seen=2 accepted=0 deadline=45s cleanup=scheduled

Source