How I gave my desktop AI companion computer use: a Cua driver, fresh element tokens, and a Rust patch for owned menus

说明与用途

Ankita 通过使用新鲜元素令牌和修复所有者窗口的 Rust 补丁，获得了点击 UI 元素的能力。

问题

Ankita, a desktop AI companion, could view UI elements but could not click them, limiting its ability to interact with desktop applications.

结果

Ankita gained the ability to click UI elements and interact with menus, thanks to fresh element tokens and a Rust patch that fixes overlay z-order for owned windows.

工作流

• 1. Keep the agent engine in Ankita and outsource input and action to the Cua Driver via MCP.
• 2. Discover executable paths, read manifests, and expose only advertised tools.
• 3. Use fresh element tokens and snapshot IDs to target UI elements, avoiding stale indices.
• 4. Verify state after actions with driver’s verify_state or read back the UI.
• 5. Connect computer use through an explicit user action, granting native desktop actions without per-action prompts.
• 6. Patch the Cua Driver to handle owned popups by aligning overlay z-order with the driven window’s ownership chain.
• 7. Build the patched driver, run overlay tests, and generate a receipt with hash and lockfile.
• 8. Deploy the patched driver and verify overlay visibility over menus in Notepad.

证据

She could not click things. Every desktop app on your machine was a place she could see on a screenshot but never touch.

来源

The driver does not call a model; it is a hand, not a brain.

来源

To type into a text box, the agent must present the fresh element_token and the current snapshot_id.

来源

Connecting computer use is an explicit user act — "Connect Computer Use" — not something that happens on startup.

来源

vendor/cua-driver/windows-owned-popups.patch (334 lines of Rust) teaches the overlay to follow the driven window's ultimate native ownership family...

来源

Before: menu z=13 topmost=true, overlay z=30 topmost=false — assertion failed as expected. After: overlay z=13 above menu z=14...

来源