Shadow Automator

说明与用途

Shadow Automator 是一个本地桌面自动化工具，使用 Qwen2.5‑VL‑3B 视觉语言模型根据文本定位 UI 元素，进行离线 PII 去除，显示 AR 覆盖，并在失败时自动修复。

问题

Millions of office tasks in desktop apps lack APIs; traditional RPA records fragile pixel coordinates that break when UI changes, and sending screenshots to cloud is unacceptable for sensitive workflows.

结果

Shadow Automator enables robust, privacy‑preserving desktop automation that adapts to UI changes, eliminates cloud dependencies, and provides real‑time ROI insights.

工作流

• Capture screen with mss and redact PII using OpenCV + regex
• Send sanitized image to local Qwen2.5‑VL‑3B via llama‑server
• Receive bounding‑box coordinates [x1,y1,x2,y2]
• Project glowing AR overlay on screen
• Click target with PyAutoGUI using easeOutQuad smoothing
• Validate UI state change with MSE; if unchanged, re‑ground and retry
• Log task to SQLite and update ROI dashboard

证据

Millions of everyday office tasks happen inside desktop apps and internal tools that expose no API

来源

Shadow Automator is a local desktop automation agent. You describe what you want in natural language via a Ctrl+Space Spotlight bar.

来源

Captures your screen (via mss) and redacts PII (credit cards, SSNs, emails) offline with OpenCV + regex

来源

Sends the sanitised screenshot to a locally-hosted Qwen2.5-VL-3B vision model

来源

Receives exact [x1, y1, x2, y2] bounding-box coordinates for the target element

来源

Clicks with human-like mouse smoothing via PyAutoGUI

来源

Logs every task to SQLite and displays live ROI savings (cloud cost + human hours avoided)

来源