Accessibility tree vs screenshots: the token math behind my browser agent

Description and use

Using the accessibility tree instead of screenshots cuts data size and token costs, improves click reliability, and limits vision model usage to only when necessary.

Problem

AI agents in browsers often rely on expensive screenshots and vision models to understand page content, leading to high token costs and unreliable clicks.

Outcome

Using the accessibility tree reduces data size by 6‑10×, cuts token costs, eliminates coordinate drift, and improves click reliability. Screenshots are reserved for canvas content or verification, keeping overall token usage low.

Workflow

• 1. Capture accessibility tree snapshot with Playwright MCP or chrome‑bridge CLI.
• 2. Parse the tree to identify target element references (e.g., @e12).
• 3. Issue a click command using the element reference via chrome‑bridge CLI or Playwright API.
• 4. Optionally take a screenshot only if the tree reports "uncertain" or if the target is a canvas.
• 5. Verify action success by comparing pre‑ and post‑action snapshots or checking for state changes.
• 6. Repeat for subsequent steps in the task.

Evidence

accessibility tree of the full page: 2.4 KB of text

Source

1000px-wide JPEG of the same page: 16 KB

Source

The tree has no guessing: an agent that clicks pixel coordinates sometimes clicks confidently wrong.

Source