Your computer-use agent clicks 20 pixels off: screenshots vs the accessibility tree

Описание и применение

Агенты, использующие только скриншоты, часто делают клики с ошибкой в 20 пикселей из‑за проблем с масштабированием. Используя сначала дерево доступности, они получают точные координаты и состояние элементов, а затем применяют OCR и скриншоты только при необходимости, что повышает точность и экономит токены.

Задача

Computer-use agents often miss target elements by about 20 pixels when relying solely on screenshots, leading to unreliable interactions.

Результат

Using the accessibility tree first eliminates off‑by‑pixel clicks, provides element state, and reduces token costs. OCR and screenshots are reserved for cases where the tree is incomplete, ensuring reliable agent interactions across platforms.

Рабочий процесс

• 1. Query the OS accessibility tree to list interactive elements with names, roles, and positions.
• 2. If the tree provides labels, let the model select an element by ID and click it directly.
• 3. If the tree lacks labels, perform OCR on the window and match visible text to identify the target.
• 4. If OCR fails (e.g., canvas or custom controls), capture a screenshot and use image analysis to locate the element.
• 5. After each action, re‑read the tree or text to verify the click succeeded.

Доказательства

A screenshot is a grid of pixels. The tree is a list of facts: button "Export" enabled @ (812, 64) textbox "File name" focused value "report-q3" menuitem "CSV" @ (830, 140)

Источник

That changes three things for an agent: It clicks the element, not a guessed coordinate. It knows state. It is cheap to send.

Источник

On a Retina Mac the screenshot is in physical pixels and clicks are in logical points (half). On Windows, display scaling of 125% or 150% does the same thing.

Источник

The order that works: 1. Tree first. 2. Text next. 3. Screenshot last. 4. Verify after acting.

Источник