Coordinate Clicks Are a Trap: Driving a GUI Agent Without Pixel Guessing

Описание и применение

GUI‑агенты, использующие координаты пикселей, часто терпят неудачу из‑за перемещения контента. Переключение на дерево доступности, клавиатурную навигацию, проверку после каждого шага и явное управление ошибками повышает надёжность.

Задача

GUI agents that click by pixel coordinates fail because UI content moves while pixels stay fixed.

Результат

Using tree‑based selection, keyboard focus, per‑step verification, and explicit error handling eliminates the brittleness of pixel‑based clicking and improves agent reliability.

Рабочий процесс

• Read the accessibility tree before clicking
• Select the node whose text is "Send" instead of using coordinates
• If the tree reports a single giant node, use keyboard navigation (TAB/arrow keys) to focus the desired element
• After each action, read the tree again to verify the expected change
• Treat an empty diff as an error and wait for a specific predicate with a timeout
• Never rely on cached screenshots; capture fresh state before each action

Доказательства

Both desktop and mobile expose an accessibility tree: uiautomator dump on Android, UI Automation on Windows, the accessibility APIs on macOS.

Источник

Clicking the node whose text is "Send" survives a re-layout; clicking (984, 183) does not.

Источник

Read → act → read. It feels wasteful. It is faster overall, because you stop discovering five steps later that step three landed on the wrong window and everything after it was theatre.

Источник