Building Real-Time Desktop Perception with Neo v0.1 Vision Engine

Описание и применение

Neo v0.1 обеспечивает реальное время восприятие рабочего стола с задержкой менее 200 мс, корректно обрабатывая высокие DPI, многомониторные и динамические интерфейсы, и обеспечивает безопасную автоматизацию через ручное подтверждение критических действий.

Задача

Desktop AI agents struggle with high‑DPI/multi‑monitor setups, dynamic application states, and non‑standard UI frameworks, leading to coordinate translation errors, missed UI elements, and unreliable automation.

Результат

Neo v0.1 achieves real‑time desktop perception with <200 ms latency, accurately handling high‑DPI, multi‑monitor, and dynamic UI scenarios, and provides secure automation through human‑approved safety gates.

Рабочий процесс

• Deploy dual‑GPU cluster with NVIDIA A10G workers
• Capture desktop frames and downsample for processing
• Apply visual grounding to locate UI elements
• Run hybrid OCR (traditional + visual context) to read text
• Reconstruct UI element tree for structured interaction
• Match user intent with fuzzy logic and typo tolerance
• Route critical actions through human approval safety gates
• Execute automation commands on target UI elements

Доказательства

Shipped critical upgrades to LuxurAI Cowork's Neo v0.1 vision engine – our in‑house vision‑language model for desktop screen perception and automation.

Источник

Dedicated NVIDIA A10G workers for <200ms latency

Источник