Solving the Resolution Gap in Anthropics' 'Computer Use': Why Normalized Coordinates Matter

说明与用途

坐标归一化对于在不同 DPI 和视口尺寸的设备上实现可靠的 LLM 驱动 GUI 交互至关重要。

问题

Implementing LLM‑driven GUI interaction requires translating coordinates across devices with varying DPI and viewports, leading to phantom misses and subpixel errors.

结果

Normalized coordinates eliminate resolution‑dependent failures, enabling reliable LLM‑driven GUI interactions across devices.

工作流

• 1. Capture raw screen coordinates from the agent’s instruction.
• 2. Retrieve current display DPI and viewport dimensions.
• 3. Apply a deterministic projection to map physical coordinates onto a 1024×768 virtual buffer.
• 4. Clamp out‑of‑bounds inputs to prevent illegal memory access.
• 5. Return normalized coordinates to the LLM for execution.

证据

An agent might perceive instructions relative to whatever environment it inhabits, but interacting with diverse hardware requires bridging highly variable physical realities with strict logical constraints.

来源

You develop an orchestration loop assuming a certain window size or density (DPI), attempt bitmapped interactions locally, and suddenly find yourself debugging phantom misses caused by subpixel offsets or incorrect bounding box calculations against much higher DPI displays later.

来源

To address exactly this discrepancy between arbitrary monitor layouts and rigid interface specifications, I developed the Claude Computer Use Coordinate Normalizer.

来源