Shadow Automator

Описание и применение

Shadow Automator – локальный инструмент автоматизации рабочего стола, использующий модель Qwen2.5‑VL‑3B для поиска UI‑элементов по тексту, с офлайн‑редакцией PII, AR‑оверлеем и самовосстановлением.

Задача

Millions of office tasks in desktop apps lack APIs; traditional RPA records fragile pixel coordinates that break when UI changes, and sending screenshots to cloud is unacceptable for sensitive workflows.

Результат

Shadow Automator enables robust, privacy‑preserving desktop automation that adapts to UI changes, eliminates cloud dependencies, and provides real‑time ROI insights.

Рабочий процесс

• Capture screen with mss and redact PII using OpenCV + regex
• Send sanitized image to local Qwen2.5‑VL‑3B via llama‑server
• Receive bounding‑box coordinates [x1,y1,x2,y2]
• Project glowing AR overlay on screen
• Click target with PyAutoGUI using easeOutQuad smoothing
• Validate UI state change with MSE; if unchanged, re‑ground and retry
• Log task to SQLite and update ROI dashboard

Доказательства

Millions of everyday office tasks happen inside desktop apps and internal tools that expose no API

Источник

Shadow Automator is a local desktop automation agent. You describe what you want in natural language via a Ctrl+Space Spotlight bar.

Источник

Captures your screen (via mss) and redacts PII (credit cards, SSNs, emails) offline with OpenCV + regex

Источник

Sends the sanitised screenshot to a locally-hosted Qwen2.5-VL-3B vision model

Источник

Receives exact [x1, y1, x2, y2] bounding-box coordinates for the target element

Источник

Clicks with human-like mouse smoothing via PyAutoGUI

Источник

Logs every task to SQLite and displays live ROI savings (cloud cost + human hours avoided)

Источник