agent-desktop 1.0: Observe-Act-Verify Loops via Windows UI Automation Instead of Screenshot Guessing

Описание и применение

Agent-desktop 1.0 позволяет агентам взаимодействовать с настольными приложениями, используя дерево доступности вместо скриншотов, что повышает надёжность и снижает потребление токенов.

Задача

Coordinate-based UI automation is fragile due to DPI scaling, themes, window position, and occlusion, causing flaky agent behavior.

Результат

Agent-desktop enables reliable, low-latency, token-efficient desktop automation by using structured accessibility trees instead of screenshots, reducing flakiness and eliminating the need for vision models.

Рабочий процесс

• Install agent-desktop via npm
• Use agent-desktop snapshot --app <AppName> -i to capture the UI tree and obtain a snapshot ID and refs
• Perform actions such as agent-desktop click <ref> using the ref
• Take another snapshot to verify the action succeeded
• Optionally use --skeleton and --compact to reduce token usage for dense Electron apps
• Handle partial observations and STALE_REF errors by retrying snapshots

Доказательства

Vercel Labs dropped agent-desktop 1.0.2 on October 30, 2026. It is a Rust CLI and C library that lets any shell-capable agent read and operate native desktop apps on Windows, macOS, and Linux by walking the OS accessibility tree instead of guessing from screenshots.

Источник

The snapshot command walks the UIA tree (or AX on macOS, AT-SPI on Linux), assigns deterministic refs in depth-first order, and embeds the snapshot id so the ref is self-describing.

Источник

The tool reports completeness on the observation and on any truncated node, so you know when the walk hit a time or depth budget.

Источник

UIA is not a perfect map of the visual tree. Many Win32 apps still expose thin or missing trees. Custom-drawn controls, games, and some Electron builds that ship with accessibility disabled will return almost nothing useful.

Источник