NexusOS — Your Personal Jarvis, Powered by AI

Description and use

NexusOS is a local AI-powered assistant that controls the computer, browser, files, and smart home devices via voice or text, using Whisper, PyAutoGUI, and Playwright.

Problem

Users want a personal assistant that can control their computer, browse the web, manage files, and interact with smart home devices via voice or text without manual input.

Outcome

NexusOS enables hands-free computer interaction, automating routine tasks, and integrating smart home control, all while preserving privacy by running locally.

Workflow

• 1. Say wake word "Hey Nexus" to activate the system.
• 2. Voice or text input is transcribed by Whisper and processed by the backend.
• 3. Depending on the command, the system uses PyAutoGUI for computer control, Playwright for browser tasks, or file system APIs for file management.
• 4. For multi-step routines, the system executes predefined sequences automatically.
• 5. Contextual memory is retrieved from SQLite vector embeddings to inform responses.
• 6. Smart home actions are sent via MQTT to Home Assistant or compatible devices.
• 7. Responses are spoken back using TTS or displayed in the UI.

Evidence

Voice Pipeline — Listens for a wake word, transcribes with Whisper, and speaks back using TTS. The whole loop takes about 2-3 seconds

Source

Computer Control — Moves your mouse, clicks buttons, and types text using PyAutoGUI

Source

Browser Automation — Built on Playwright, it can open browsers, navigate, fill forms, and read page content back to you

Source