AI Agent Email Probes Need a Test Contract

Description and use

Implementing a bounded, evidence‑based email probe eliminates flaky verification steps and improves test reliability.

Problem

AI agents struggle with unreliable inboxes, leading to false positives in automated workflows.

Outcome

Implementing a bounded, evidence‑based email probe eliminates flaky verification steps and improves test reliability.

Workflow

• Create a mailbox record with run_id, mailbox, created_at, expires_at, expected_recipient, seen_message_ids
• Ask the application to send a verification message
• Poll the inbox for messages newer than created_at, filtering by expected_recipient and unseen message IDs
• If a message matches the correlation token, record evidence and transition to FOUND
• Return evidence with status, message_id, recipient, received_at, age_seconds, correlation_match
• If deadline expires, return evidence with status EXPIRED and attempts count
• Cleanup mailbox after VERIFIED, INVALID, or EXPIRED, ensuring idempotent release

Evidence

AI agents are good at following a workflow such as “sign up, open the verification email, and finish setup.” They are much less good at dealing with an inbox that is slow, empty, duplicated, or shared with another test.

Source

Without explicit boundaries, the agent can make a plausible choice and still produce a false positive.

Source

The probe does not click a link or mutate application state. That separation is useful: one component observes, another component performs the next action, and the agent can explain which evidence allowed it to proceed.

Source