agent-desktop 1.0: Observe-Act-Verify Loops via Windows UI Automation Instead of Screenshot Guessing

说明与用途

Agent-desktop 1.0 通过使用可访问性树而非截图，让代理能够更可靠地与桌面应用交互，降低令牌消耗。

问题

Coordinate-based UI automation is fragile due to DPI scaling, themes, window position, and occlusion, causing flaky agent behavior.

结果

Agent-desktop enables reliable, low-latency, token-efficient desktop automation by using structured accessibility trees instead of screenshots, reducing flakiness and eliminating the need for vision models.

工作流

• Install agent-desktop via npm
• Use agent-desktop snapshot --app <AppName> -i to capture the UI tree and obtain a snapshot ID and refs
• Perform actions such as agent-desktop click <ref> using the ref
• Take another snapshot to verify the action succeeded
• Optionally use --skeleton and --compact to reduce token usage for dense Electron apps
• Handle partial observations and STALE_REF errors by retrying snapshots

证据

Vercel Labs dropped agent-desktop 1.0.2 on October 30, 2026. It is a Rust CLI and C library that lets any shell-capable agent read and operate native desktop apps on Windows, macOS, and Linux by walking the OS accessibility tree instead of guessing from screenshots.

来源

The snapshot command walks the UIA tree (or AX on macOS, AT-SPI on Linux), assigns deterministic refs in depth-first order, and embeds the snapshot id so the ref is self-describing.

来源

The tool reports completeness on the observation and on any truncated node, so you know when the walk hit a time or depth budget.

来源

UIA is not a perfect map of the visual tree. Many Win32 apps still expose thin or missing trees. Custom-drawn controls, games, and some Electron builds that ship with accessibility disabled will return almost nothing useful.

来源