Coordinate Clicks Are a Trap: Driving a GUI Agent Without Pixel Guessing

说明与用途

使用像素坐标点击的GUI代理因UI内容移动而经常失败。改用可访问性树选择、键盘导航、每步验证和显式错误处理可提升可靠性。

问题

GUI agents that click by pixel coordinates fail because UI content moves while pixels stay fixed.

结果

Using tree‑based selection, keyboard focus, per‑step verification, and explicit error handling eliminates the brittleness of pixel‑based clicking and improves agent reliability.

工作流

• Read the accessibility tree before clicking
• Select the node whose text is "Send" instead of using coordinates
• If the tree reports a single giant node, use keyboard navigation (TAB/arrow keys) to focus the desired element
• After each action, read the tree again to verify the expected change
• Treat an empty diff as an error and wait for a specific predicate with a timeout
• Never rely on cached screenshots; capture fresh state before each action

证据

Both desktop and mobile expose an accessibility tree: uiautomator dump on Android, UI Automation on Windows, the accessibility APIs on macOS.

来源

Clicking the node whose text is "Send" survives a re-layout; clicking (984, 183) does not.

来源

Read → act → read. It feels wasteful. It is faster overall, because you stop discovering five steps later that step three landed on the wrong window and everything after it was theatre.

来源