Your AI Agent Can't See What It Clicks

说明与用途

桌面自动化代理因 UI 变化和 HiDPI 缩放而经常误点击。捕获屏幕截图、绘制缩放注释并将其返回给代理可防止错误操作。

问题

AI agents that drive a desktop by moving the mouse and typing keystrokes operate blind, often clicking empty space or wrong fields when the UI shifts, a dialog steals focus, or HiDPI scaling changes the math.

结果

Agents now have a visual baseline, reducing misclicks and enabling safe automation across multi‑monitor and HiDPI setups.

工作流

• Capture current screen using mss to get a Pillow RGB image.
• Determine OS scaling factor (e.g., via Windows API or macOS methods).
• Create Annotation objects with logical coordinates and desired styles.
• Scale annotations by the detected scale factor and draw them onto the screenshot using PIL.
• Feed the annotated image back to the agent for visual confirmation or further processing.

证据

AI agents that drive a desktop by moving the mouse and typing keystrokes operate blind.

来源

Grab the frame, then decide. This also gives you a baseline image to annotate after the action.

来源

On Windows and macOS, the logical coordinate space (what pyautogui returns) differs from the physical pixel space (what the screenshot contains).

来源