The accessibility tree gave 162 refs before any screenshot

Description and use

agent-desktop retrieved 162 interactive elements from Finder via macOS accessibility tree without screenshots, showing a safer, rejection‑aware automation method.

Problem

Desktop automation agents rely on pixel-based coordinate math, which can fail silently after UI changes.

Outcome

The tool successfully retrieved 162 interactive elements from Finder via the accessibility tree without screenshots, demonstrating a safer, rejection‑aware approach compared to pixel math.

Workflow

• npm install -g agent-desktop
• agent-desktop version
• agent-desktop permissions --guide
• agent-desktop list-apps
• agent-desktop snapshot --app Finder -i
• agent-desktop screenshot --app Finder <path>

Evidence

agent-desktop snapshot --app Finder -i returned ok=true, error=null, ref_count=162 interactive elements, complete=false, window title Finder

Source

The permission census from agent-desktop permissions --guide read accessibility GRANTED (already on for my terminal automation chain), automation UNKNOWN, screen_recording DENIED

Source

agent-desktop list-apps returned zero apps headless, consistent with their documented prohibition on foreground activation for headless calls

Source