Improving Desktop Screen Perception with Neo v0.1 Vision Engine

Description and use

Neo v0.1 improves desktop automation by providing accurate visual grounding and OCR in multi‑monitor high‑DPI setups, enabling reliable interaction with UI elements.

Problem

Desktop automation requires accurate visual grounding and OCR text detection, especially in multi-monitor high‑DPI setups.

Outcome

Neo v0.1 enhances LuxurAI Cowork’s desktop automation by providing reliable visual grounding and OCR in multi‑monitor high‑DPI setups, enabling accurate UI interactions at 15 fps.

Workflow

• Capture live desktop frames at 15 fps using dual‑GPU cluster
• Apply Windows UI Automation (UIA) to identify UI elements
• Use low‑level Win32 APIs to interact with applications (type, click)
• Perform OCR on captured frames for text detection
• Integrate results into LuxurAI Cowork automation workflows

Evidence

Neo v0.1 runs on our dedicated dual‑GPU cluster, processing live desktop frames at 15 fps.

Source

Desktop automation requires accurate visual grounding and OCR text detection, especially in multi-monitor high-DPI setups.

Source