Building PhotoCraft Companion: Local AI for a WebAssembly Editor (Rust Architecture Deep Dive)

说明与用途

一个基于 Rust 的独立伴侣进程通过 JSON‑RPC 与 PhotoCraft 通信，并通过剪贴板交换像素缓冲区，实现本地神经网络推理、GPU 加速和浮动 UI，无需遥测。

问题

Running modern neural networks locally in a WebAssembly sandbox causes UI freezes, memory limits, and no floating UI, forcing users to rely on cloud services.

结果

The companion pattern delivers local AI inference with GPU acceleration, a floating UI, and zero‑telemetry, eliminating cloud dependencies and improving workflow latency.

工作流

• 1. Design a Rust companion that runs as a separate OS process.
• 2. Use PhotoCraft’s JSON‑RPC control socket (TCP 18492) for command exchange.
• 3. Transfer pixel buffers via the system clipboard to avoid Wasm memory limits.
• 4. Implement a transactional lock around the TCP client to serialize request/response cycles.
• 5. Add a non‑blocking poller for brush color to avoid contention.
• 6. Build a floating overlay with egui/eframe that stays above the canvas.
• 7. Package models and manifests in JSON, keeping the core zero‑hardcoded.
• 8. Deploy the companion as a single binary with system‑tray integration.

证据

No direct GPU access. No DirectML, no CoreML, no CUDA or Tensor Cores. Everything runs on the CPU through Wasmtime.

来源

Loading multi-hundred-megabyte model weights into a browser-grade or sandboxed Wasm linear memory pool quickly hits execution ceilings.

来源

The companion is structured as a Cargo workspace with 6 crates, ~25,000 lines of Rust, and governed by strict engineering constraints: a Definition of Done, resource guards, automated architectural linters, and bilingual localization enforced by CI.

来源