Gemini Computer Use GA: Screenshot Loops, Coordinate Grids, and Why Accessibility Trees Still Win on Windows Latency

Description and use

Screenshot‑based Gemini Computer Use offers cross‑platform support but incurs high latency (400‑800 ms). UI Automation provides low latency and high reliability but lacks custom UI support. A hybrid approach using UIA with screenshot fallbacks delivers the best trade‑off for desktop agents.

Problem

Real‑time Windows agents need low‑latency UI interaction but screenshot‑based Computer Use incurs 400‑800 ms per step, making it unsuitable for live overlays.

Outcome

Screenshot‑based Computer Use offers cross‑platform and custom‑UI support but suffers high latency; UIA provides low latency and high reliability on standard Win32/WPF but lacks support for custom‑drawn interfaces. A hybrid strategy—prefer UIA, fallback to screenshots—yields the best trade‑off for production desktop agents.

Workflow

• Capture screenshot (GDI BitBlt or DXGI duplication)
• Encode to base64 and send to Gemini via Interactions API
• Receive structured action (click, type, scroll) with 0‑999 normalized coordinates
• Scale coordinates to actual screen size
• Execute action via Win32 SendInput or UIA Invoke
• Verify post‑condition (control state change)
• Loop until task completion or safety filter triggers

Evidence

On October 30, 2026 Google flipped Computer Use to general availability across both the Interactions API and GenerateContent.

Source

Coordinates come back on a 0-999 normalized grid. You scale them to the real screen or window.

Source

On a local machine that can easily be 400-800 ms per step before the UI even reacts.

Source

The alternative that Windows has had for years is UI Automation (UIA).

Source