Your computer-use agent clicks 20 pixels off: screenshots vs the accessibility tree

说明与用途

仅使用截图的代理经常因缩放问题而误点约20像素。先查询可访问性树获取精确坐标和状态，再在必要时回退到OCR和截图，可提升准确性并降低令牌消耗。

问题

Computer-use agents often miss target elements by about 20 pixels when relying solely on screenshots, leading to unreliable interactions.

结果

Using the accessibility tree first eliminates off‑by‑pixel clicks, provides element state, and reduces token costs. OCR and screenshots are reserved for cases where the tree is incomplete, ensuring reliable agent interactions across platforms.

工作流

• 1. Query the OS accessibility tree to list interactive elements with names, roles, and positions.
• 2. If the tree provides labels, let the model select an element by ID and click it directly.
• 3. If the tree lacks labels, perform OCR on the window and match visible text to identify the target.
• 4. If OCR fails (e.g., canvas or custom controls), capture a screenshot and use image analysis to locate the element.
• 5. After each action, re‑read the tree or text to verify the click succeeded.

证据

A screenshot is a grid of pixels. The tree is a list of facts: button "Export" enabled @ (812, 64) textbox "File name" focused value "report-q3" menuitem "CSV" @ (830, 140)

来源

That changes three things for an agent: It clicks the element, not a guessed coordinate. It knows state. It is cheap to send.

来源

On a Retina Mac the screenshot is in physical pixels and clicks are in logical points (half). On Windows, display scaling of 125% or 150% does the same thing.

来源

The order that works: 1. Tree first. 2. Text next. 3. Screenshot last. 4. Verify after acting.

来源