AI Agent Email Probes Need a Test Contract

说明与用途

通过实现受限、基于证据的邮件探针，消除了自动化测试中的不稳定性，提高了测试可靠性。

问题

AI agents struggle with unreliable inboxes, leading to false positives in automated workflows.

结果

Implementing a bounded, evidence‑based email probe eliminates flaky verification steps and improves test reliability.

工作流

• Create a mailbox record with run_id, mailbox, created_at, expires_at, expected_recipient, seen_message_ids
• Ask the application to send a verification message
• Poll the inbox for messages newer than created_at, filtering by expected_recipient and unseen message IDs
• If a message matches the correlation token, record evidence and transition to FOUND
• Return evidence with status, message_id, recipient, received_at, age_seconds, correlation_match
• If deadline expires, return evidence with status EXPIRED and attempts count
• Cleanup mailbox after VERIFIED, INVALID, or EXPIRED, ensuring idempotent release

证据

AI agents are good at following a workflow such as “sign up, open the verification email, and finish setup.” They are much less good at dealing with an inbox that is slow, empty, duplicated, or shared with another test.

来源

Without explicit boundaries, the agent can make a plausible choice and still produce a false positive.

来源

The probe does not click a link or mutate application state. That separation is useful: one component observes, another component performs the next action, and the agent can explain which evidence allowed it to proceed.

来源