Building Neo v0.1: Dual-GPU Vision Engine for Desktop Automation

Description and use

Neo v0.1 provides accurate UI detection and OCR for desktop automation on high‑DPI displays, leveraging a dual‑GPU cluster and a safety overlay.

Problem

Desktop automation needs precise screen perception, but most solutions struggle with high‑DPI displays and multi‑monitor setups.

Outcome

Neo v0.1 delivers reliable, low‑latency desktop automation on high‑DPI, multi‑monitor setups, with safety safeguards for critical actions.

Workflow

• Capture desktop frames live
• Detect UI elements and OCR text
• Map detected elements to Windows UI Automation actions
• Execute tasks via UIA
• If action is high‑risk, present safety overlay for human approval

Evidence

Neo v0.1 processes desktop frames at 15 fps, detects UI elements, and uses Windows UI Automation (UIA) to execute tasks.

Source