Your computer-use agent clicks 20 pixels off: screenshots vs the accessibility tree

Description and use

Agents relying solely on screenshots frequently miss targets by about 20 pixels due to scaling issues. By first querying the accessibility tree for exact coordinates and state, then falling back to OCR and screenshots only when needed, accuracy improves and token usage decreases.

Problem

Computer-use agents often miss target elements by about 20 pixels when relying solely on screenshots, leading to unreliable interactions.

Outcome

Using the accessibility tree first eliminates off‑by‑pixel clicks, provides element state, and reduces token costs. OCR and screenshots are reserved for cases where the tree is incomplete, ensuring reliable agent interactions across platforms.

Workflow

• 1. Query the OS accessibility tree to list interactive elements with names, roles, and positions.
• 2. If the tree provides labels, let the model select an element by ID and click it directly.
• 3. If the tree lacks labels, perform OCR on the window and match visible text to identify the target.
• 4. If OCR fails (e.g., canvas or custom controls), capture a screenshot and use image analysis to locate the element.
• 5. After each action, re‑read the tree or text to verify the click succeeded.

Evidence

A screenshot is a grid of pixels. The tree is a list of facts: button "Export" enabled @ (812, 64) textbox "File name" focused value "report-q3" menuitem "CSV" @ (830, 140)

Source

That changes three things for an agent: It clicks the element, not a guessed coordinate. It knows state. It is cheap to send.

Source

On a Retina Mac the screenshot is in physical pixels and clicks are in logical points (half). On Windows, display scaling of 125% or 150% does the same thing.

Source

The order that works: 1. Tree first. 2. Text next. 3. Screenshot last. 4. Verify after acting.

Source