Email Automation Needs a Dry-Run Contract

Description and use

Introducing a dry‑run contract in email automation tests explicitly defines expected actions, validates them before execution, and provides concrete failure states, improving debugging and reliability.

Problem

Email automation tests often fail with vague timeout errors, leaving unclear which resource or step caused the failure.

Outcome

Introducing a dry‑run contract clarifies expected side effects, enables early validation, and produces explicit failure states that improve debugging and reliability of email automation tests.

Workflow

• Define a dry‑run manifest with run_id, case, mailbox (kind, lease_seconds), expected (messages, subject_contains), and cleanup.
• Validate the manifest before contacting the provider.
• Reserve the mailbox and record its address.
• Request the verification email and wait for a message with a known correlation key.
• Consume the message and validate its content against the expected subject.
• Release temporary resources according to the cleanup policy.
• Report the outcome with explicit failure states such as configuration_error, delivery_timeout, assertion_failed, or cleanup_failed.

Evidence

Email automation often fails before the email provider is actually the problem. A test creates a user, requests a verification message, waits for a mailbox, and then reports only “timeout.”

Source

I like treating email automation as a small workflow with a dry‑run contract. Before the test sends anything, it can describe the mailbox it wants, the messages it expects, and the evidence it must collect.

Source

The runner can validate the manifest before it contacts the provider: ... raise ValueError if missing fields or invalid lease.

Source

Every transition should be idempotent. If the CI worker retries after a network timeout, it should look up the existing run_id before reserving another mailbox.

Source