Accessibility tree vs screenshots: the token math behind my browser agent

说明与用途

使用可访问性树而非截图可降低数据量和令牌成本，提高点击可靠性，仅在必要时使用视觉模型。

问题

AI agents in browsers often rely on expensive screenshots and vision models to understand page content, leading to high token costs and unreliable clicks.

结果

Using the accessibility tree reduces data size by 6‑10×, cuts token costs, eliminates coordinate drift, and improves click reliability. Screenshots are reserved for canvas content or verification, keeping overall token usage low.

工作流

• 1. Capture accessibility tree snapshot with Playwright MCP or chrome‑bridge CLI.
• 2. Parse the tree to identify target element references (e.g., @e12).
• 3. Issue a click command using the element reference via chrome‑bridge CLI or Playwright API.
• 4. Optionally take a screenshot only if the tree reports "uncertain" or if the target is a canvas.
• 5. Verify action success by comparing pre‑ and post‑action snapshots or checking for state changes.
• 6. Repeat for subsequent steps in the task.

证据

accessibility tree of the full page: 2.4 KB of text

来源

1000px-wide JPEG of the same page: 16 KB

来源

The tree has no guessing: an agent that clicks pixel coordinates sometimes clicks confidently wrong.

来源