CI Email Tests Need a Run Manifest

Описание и применение

Добавление манифеста к тестам электронной почты в CI превращает неопределённые тайм‑ауты в подробные, полезные логи, ускоряя отладку и предотвращая загрязнение тестов.

Задача

Email assertions in CI pipelines lack actionable context, making flaky tests hard to debug.

Результат

Adding a run manifest to email tests transforms vague timeouts into detailed, actionable logs, reducing debugging time and preventing test pollution.

Рабочий процесс

• Create a unique run ID before triggering the application action.
• Record the test name, inbox address, and shard in the manifest.
• Set trigger time, poll deadline, and cleanup deadline timestamps.
• Append seen message IDs and the accepted message ID as they are processed.
• Close the manifest in a finally block, marking status as passed, failed, or expired.
• Save the manifest as an artifact and print a concise summary line on failure.
• Use the manifest to investigate delivery, routing, subject, sender, or link extraction issues.
• Implement an idempotent sweeper to clean up inboxes after expiry or job termination.
• Log decisions such as timestamp filtering or inbox removal in the manifest notes.

Доказательства

Email assertions are often the least explainable part of an otherwise tidy CI pipeline.

Источник

I have started treating each email test like a small deployment. It gets an identity, a start time, a cleanup deadline, and a receipt.

Источник

A useful manifest does not need to become a new database. Four groups of fields cover most debugging sessions: Ownership, Timing, Matching, Lifecycle.

Источник

When a test fails, print a one-line summary and save the full manifest as an artifact: email-run signup-1842-attempt-1 failed: seen=2 accepted=0 deadline=45s cleanup=scheduled

Источник