Solving the Resolution Gap in Anthropics' 'Computer Use': Why Normalized Coordinates Matter

Описание и применение

Для корректного взаимодействия LLM с графическим интерфейсом необходимо нормализовать координаты, чтобы устранить ошибки, связанные с разной плотностью пикселей и размерами экранов.

Задача

Implementing LLM‑driven GUI interaction requires translating coordinates across devices with varying DPI and viewports, leading to phantom misses and subpixel errors.

Результат

Normalized coordinates eliminate resolution‑dependent failures, enabling reliable LLM‑driven GUI interactions across devices.

Рабочий процесс

• 1. Capture raw screen coordinates from the agent’s instruction.
• 2. Retrieve current display DPI and viewport dimensions.
• 3. Apply a deterministic projection to map physical coordinates onto a 1024×768 virtual buffer.
• 4. Clamp out‑of‑bounds inputs to prevent illegal memory access.
• 5. Return normalized coordinates to the LLM for execution.

Доказательства

An agent might perceive instructions relative to whatever environment it inhabits, but interacting with diverse hardware requires bridging highly variable physical realities with strict logical constraints.

Источник

You develop an orchestration loop assuming a certain window size or density (DPI), attempt bitmapped interactions locally, and suddenly find yourself debugging phantom misses caused by subpixel offsets or incorrect bounding box calculations against much higher DPI displays later.

Источник

To address exactly this discrepancy between arbitrary monitor layouts and rigid interface specifications, I developed the Claude Computer Use Coordinate Normalizer.

Источник