AI Test Agents Need a Mailbox Lease

Description and use

Mailbox lease modeling isolates test state, prevents false positives, and ensures reproducible evidence for AI-driven test agents.

Problem

AI test agents struggle with shared mailbox state, leading to false positives, token leakage, and cleanup failures.

Outcome

Implementing mailbox leases eliminates shared state issues, reduces false passes, and provides clear evidence for debugging.

Workflow

• Allocate a mailbox and create an active lease before starting the test
• Stamp the run_id in the test user’s email address or metadata
• Trigger the signup or notification action
• Poll for messages newer than the lease creation time using a cursor
• Validate sender, recipient, subject, event type, and token destination
• Record a compact receipt with message ID, timestamps, and assertion result
• Release or expire the mailbox in a finally‑style cleanup step

Evidence

An agent that uses one shared inbox quickly inherits hidden state: old messages, another job's verification link, rate limits, and a cleanup rule nobody wrote down.

Source

I like to model each test mailbox as a lease. The agent receives a mailbox for one run, knows what it is allowed to do, and must return or expire it when the run ends.

Source

The workflow can be implemented with ordinary API calls. The important part is the order: Allocate, Stamp, Trigger, Poll, Validate, Record, Release.

Source