I built the assistant I clicked on as a kid. It haunts me now!

Description and use

Paperclip is a local macOS desktop assistant that watches the active window, runs a Gemma 4 vision model via Ollama, persists state with Temporal, and never sends screenshots outside the machine.

Problem

A user wanted a desktop assistant that could observe the screen locally, offer help, and persist across restarts without sending data externally.

Outcome

Paperclip provides a local, persistent, privacy‑preserving desktop assistant that observes the screen, offers context‑aware help, and survives restarts without external data leakage.

Workflow

• Clone repo
• Run npm run setup to install Temporal, Ollama, and dependencies
• Configure .env and macOS permissions
• Start ghost with npm run ghost:start
• Ghost checks active window every 5 s
• When window changes, take screenshot, resize, run Gemma 4 via Ollama
• Parse JSON output {app,activity,detail,notable}
• If stuck >20 min, offer help
• Persist state in Temporal workflow
• Restart ghost via launchd if killed

Evidence

Paperclip runs a local vision model through Ollama, so nothing about your screen leaves your machine.

Source

Each glance is one invoke_agent span in Sentry, with every step nested under it.

Source

The biggest speed win came from asking for less. Instead of a paragraph, the model now returns a small JSON object: {app, activity, detail, notable}.

Source