Agent Replay Caching for E2E Tests: How TesterArmy Records Once, Replays Until the App Changes

Description and use

TesterArmy records agent actions once and replays them until the app changes, reducing token cost and improving test determinism.

Problem

Agent‑driven E2E tests are expensive and flaky because each run calls a language model and the model can produce different action sequences.

Outcome

The replay cache reduces token cost to zero after the first run and makes tests deterministic until the app changes, while still allowing manual assertions to detect stale caches.

Workflow

• Write a natural‑language goal with agent.act()
• Framework records the resulting action sequence and locator snapshots on first run
• Store the sequence in a JSON cache file per test
• On later runs, check each locator against the current page
• If all locators resolve, replay cached actions
• If any locator fails, invalidate cache and call the agent again

Evidence

TesterArmy's E2E framework solves the two hard problems of agent-driven testing: cost and flakiness.

Source

On the first run, the framework records every action the agent takes. On subsequent runs, it replays those actions without calling the model until the app changes enough to invalidate the recording.

Source

The framework does not store DOM snapshots. It stores action sequences and the locators that triggered them.

Source

If an assertion fails after a replayed action, the test fails, and you know the cache is stale.

Source