Accessibility tree vs screenshots: the token math behind my browser agent

Описание и применение

Использование дерева доступности вместо скриншотов снижает размер данных и токенов, повышает надёжность кликов и уменьшает расходы на модели зрения.

Задача

AI agents in browsers often rely on expensive screenshots and vision models to understand page content, leading to high token costs and unreliable clicks.

Результат

Using the accessibility tree reduces data size by 6‑10×, cuts token costs, eliminates coordinate drift, and improves click reliability. Screenshots are reserved for canvas content or verification, keeping overall token usage low.

Рабочий процесс

• 1. Capture accessibility tree snapshot with Playwright MCP or chrome‑bridge CLI.
• 2. Parse the tree to identify target element references (e.g., @e12).
• 3. Issue a click command using the element reference via chrome‑bridge CLI or Playwright API.
• 4. Optionally take a screenshot only if the tree reports "uncertain" or if the target is a canvas.
• 5. Verify action success by comparing pre‑ and post‑action snapshots or checking for state changes.
• 6. Repeat for subsequent steps in the task.

Доказательства

accessibility tree of the full page: 2.4 KB of text

Источник

1000px-wide JPEG of the same page: 16 KB

Источник

The tree has no guessing: an agent that clicks pixel coordinates sometimes clicks confidently wrong.

Источник