A Run Manifest for AI Email Automation

Description and use

A guide to creating a run manifest for AI‑powered email verification, defining unique run IDs, mailbox leases, atomic message claims, and budget limits to reduce flaky CI tests.

Problem

Email verification tests are flaky because mailbox state, message timing, and tool usage are hidden in prompts and temporary variables, making failures hard to reproduce and debug.

Outcome

Implementing a run manifest reduces flaky email verification tests, provides clear failure diagnostics, and enables consistent CI behavior across parallel jobs.

Workflow

• Create lease with unique run_id and mailbox address
• Send verification request to service
• Poll inbox within deadline until message matches expected sender and subject
• Atomically claim the message
• Verify token from message
• Release mailbox lease
• Publish redacted receipt as CI artifact

Evidence

An AI agent can create a test account, request a verification message, and report success in a single conversation.

Source

Without a manifest, the setup are often hidden in prompt text and temporary variables.

Source

The manifest is not a prompt. The runtime should enforce it even when an agent asks for another poll.

Source

Publish a redacted receipt as an artifact

Source