agent-desktop 1.0: Observe-Act-Verify Loops via Windows UI Automation Instead of Screenshot Guessing

Description and use

Agent-desktop 1.0 enables agents to interact with desktop apps via accessibility trees instead of screenshots, improving reliability and reducing token usage.

Problem

Coordinate-based UI automation is fragile due to DPI scaling, themes, window position, and occlusion, causing flaky agent behavior.

Outcome

Agent-desktop enables reliable, low-latency, token-efficient desktop automation by using structured accessibility trees instead of screenshots, reducing flakiness and eliminating the need for vision models.

Workflow

• Install agent-desktop via npm
• Use agent-desktop snapshot --app <AppName> -i to capture the UI tree and obtain a snapshot ID and refs
• Perform actions such as agent-desktop click <ref> using the ref
• Take another snapshot to verify the action succeeded
• Optionally use --skeleton and --compact to reduce token usage for dense Electron apps
• Handle partial observations and STALE_REF errors by retrying snapshots

Evidence

Vercel Labs dropped agent-desktop 1.0.2 on October 30, 2026. It is a Rust CLI and C library that lets any shell-capable agent read and operate native desktop apps on Windows, macOS, and Linux by walking the OS accessibility tree instead of guessing from screenshots.

Source

The snapshot command walks the UIA tree (or AX on macOS, AT-SPI on Linux), assigns deterministic refs in depth-first order, and embeds the snapshot id so the ref is self-describing.

Source

The tool reports completeness on the observation and on any truncated node, so you know when the walk hit a time or depth budget.

Source

UIA is not a perfect map of the visual tree. Many Win32 apps still expose thin or missing trees. Custom-drawn controls, games, and some Electron builds that ship with accessibility disabled will return almost nothing useful.

Source