Gemini Computer Use GA: Screenshot Loops, Coordinate Grids, and Why Accessibility Trees Still Win on Windows Latency

说明与用途

Gemini Computer Use 的截图路径支持跨平台，但延迟高（400-800 ms）。UI Automation 提供低延迟和高可靠性，但不支持自定义 UI。混合方法结合 UIA 与截图回退，为桌面代理提供最佳权衡。

问题

Real‑time Windows agents need low‑latency UI interaction but screenshot‑based Computer Use incurs 400‑800 ms per step, making it unsuitable for live overlays.

结果

Screenshot‑based Computer Use offers cross‑platform and custom‑UI support but suffers high latency; UIA provides low latency and high reliability on standard Win32/WPF but lacks support for custom‑drawn interfaces. A hybrid strategy—prefer UIA, fallback to screenshots—yields the best trade‑off for production desktop agents.

工作流

• Capture screenshot (GDI BitBlt or DXGI duplication)
• Encode to base64 and send to Gemini via Interactions API
• Receive structured action (click, type, scroll) with 0‑999 normalized coordinates
• Scale coordinates to actual screen size
• Execute action via Win32 SendInput or UIA Invoke
• Verify post‑condition (control state change)
• Loop until task completion or safety filter triggers

证据

On October 30, 2026 Google flipped Computer Use to general availability across both the Interactions API and GenerateContent.

来源

Coordinates come back on a 0-999 normalized grid. You scale them to the real screen or window.

来源

On a local machine that can easily be 400-800 ms per step before the UI even reacts.

来源

The alternative that Windows has had for years is UI Automation (UIA).

来源