Coordinate Clicks Are a Trap: Driving a GUI Agent Without Pixel Guessing

Description and use

GUI agents that click by pixel coordinates fail because UI content moves while pixels stay fixed. Switching to accessibility tree selection, keyboard navigation, per‑step verification, and explicit error handling improves reliability.

Problem

GUI agents that click by pixel coordinates fail because UI content moves while pixels stay fixed.

Outcome

Using tree‑based selection, keyboard focus, per‑step verification, and explicit error handling eliminates the brittleness of pixel‑based clicking and improves agent reliability.

Workflow

• Read the accessibility tree before clicking
• Select the node whose text is "Send" instead of using coordinates
• If the tree reports a single giant node, use keyboard navigation (TAB/arrow keys) to focus the desired element
• After each action, read the tree again to verify the expected change
• Treat an empty diff as an error and wait for a specific predicate with a timeout
• Never rely on cached screenshots; capture fresh state before each action

Evidence

Both desktop and mobile expose an accessibility tree: uiautomator dump on Android, UI Automation on Windows, the accessibility APIs on macOS.

Source

Clicking the node whose text is "Send" survives a re-layout; clicking (984, 183) does not.

Source

Read → act → read. It feels wasteful. It is faster overall, because you stop discovering five steps later that step three landed on the wrong window and everything after it was theatre.

Source