Building PhotoCraft Companion: Local AI for a WebAssembly Editor (Rust Architecture Deep Dive)

Description and use

A Rust companion process connects to PhotoCraft via JSON‑RPC and exchanges pixel buffers through the clipboard, enabling local neural‑network inference with GPU acceleration and a floating UI without telemetry.

Problem

Running modern neural networks locally in a WebAssembly sandbox causes UI freezes, memory limits, and no floating UI, forcing users to rely on cloud services.

Outcome

The companion pattern delivers local AI inference with GPU acceleration, a floating UI, and zero‑telemetry, eliminating cloud dependencies and improving workflow latency.

Workflow

• 1. Design a Rust companion that runs as a separate OS process.
• 2. Use PhotoCraft’s JSON‑RPC control socket (TCP 18492) for command exchange.
• 3. Transfer pixel buffers via the system clipboard to avoid Wasm memory limits.
• 4. Implement a transactional lock around the TCP client to serialize request/response cycles.
• 5. Add a non‑blocking poller for brush color to avoid contention.
• 6. Build a floating overlay with egui/eframe that stays above the canvas.
• 7. Package models and manifests in JSON, keeping the core zero‑hardcoded.
• 8. Deploy the companion as a single binary with system‑tray integration.

Evidence

No direct GPU access. No DirectML, no CoreML, no CUDA or Tensor Cores. Everything runs on the CPU through Wasmtime.

Source

Loading multi-hundred-megabyte model weights into a browser-grade or sandboxed Wasm linear memory pool quickly hits execution ceilings.

Source

The companion is structured as a Cargo workspace with 6 crates, ~25,000 lines of Rust, and governed by strict engineering constraints: a Definition of Done, resource guards, automated architectural linters, and bilingual localization enforced by CI.

Source