Your AI Agent Can't See What It Clicks

Description and use

Desktop automation agents often misclick due to UI changes and HiDPI scaling. Capturing a screenshot, drawing scaled annotations, and feeding it back to the agent prevents wrong actions.

Problem

AI agents that drive a desktop by moving the mouse and typing keystrokes operate blind, often clicking empty space or wrong fields when the UI shifts, a dialog steals focus, or HiDPI scaling changes the math.

Outcome

Agents now have a visual baseline, reducing misclicks and enabling safe automation across multi‑monitor and HiDPI setups.

Workflow

• Capture current screen using mss to get a Pillow RGB image.
• Determine OS scaling factor (e.g., via Windows API or macOS methods).
• Create Annotation objects with logical coordinates and desired styles.
• Scale annotations by the detected scale factor and draw them onto the screenshot using PIL.
• Feed the annotated image back to the agent for visual confirmation or further processing.

Evidence

AI agents that drive a desktop by moving the mouse and typing keystrokes operate blind.

Source

Grab the frame, then decide. This also gives you a baseline image to annotate after the action.

Source

On Windows and macOS, the logical coordinate space (what pyautogui returns) differs from the physical pixel space (what the screenshot contains).

Source