Your AI Agent Can't See What It Clicks

Описание и применение

Автоматизированные агенты, действующие по координатам, часто ошибаются из‑за изменения UI и масштабирования HiDPI. Снимок экрана, масштабированные аннотации и их подача обратно в агент позволяют избежать неверных кликов.

Задача

AI agents that drive a desktop by moving the mouse and typing keystrokes operate blind, often clicking empty space or wrong fields when the UI shifts, a dialog steals focus, or HiDPI scaling changes the math.

Результат

Agents now have a visual baseline, reducing misclicks and enabling safe automation across multi‑monitor and HiDPI setups.

Рабочий процесс

• Capture current screen using mss to get a Pillow RGB image.
• Determine OS scaling factor (e.g., via Windows API or macOS methods).
• Create Annotation objects with logical coordinates and desired styles.
• Scale annotations by the detected scale factor and draw them onto the screenshot using PIL.
• Feed the annotated image back to the agent for visual confirmation or further processing.

Доказательства

AI agents that drive a desktop by moving the mouse and typing keystrokes operate blind.

Источник

Grab the frame, then decide. This also gives you a baseline image to annotate after the action.

Источник

On Windows and macOS, the logical coordinate space (what pyautogui returns) differs from the physical pixel space (what the screenshot contains).

Источник