Running LLMs inside a Chrome extension: WebLLM, transformers.js and the MV3 traps

说明与用途

AWFlow Chrome 扩展使用 WebLLM、transformers.js 和 TensorFlow.js 在本地运行 LLM，使用专用 Web Worker 进行推理，并支持 WebGPU，若不可用则回退到 WASM。

问题

Run AI nodes in a Chrome extension without an API key and without sending page data outside the machine.

结果

The extension successfully runs local LLMs in the browser, keeping data local, providing a fallback to WASM for unsupported hardware, and maintaining UI responsiveness by isolating heavy inference in Web Workers.

工作流

• 1. Ship three runtimes: WebLLM, transformers.js, TensorFlow.js. 2. Use dedicated Web Workers for each runtime to keep the UI thread responsive. 3. Route chat-style tasks to the WebLLM worker. 4. Route task pipelines (summarization, classification, embeddings) to a second worker shared by transformers.js and TensorFlow.js. 5. Detect WebGPU availability once; if present, use GPU for supported tasks, otherwise fall back to WASM. 6. Wrap WebLLM in a worker using LangChain’s ChatWebLLM, swapping in a worker engine when available. 7. If the worker path fails (no WebGPU, download error), fall back to the main‑thread engine without breaking the node.

证据

I wanted that step to work without an API key and without sending the page anywhere, which meant running the model inside the extension itself.

来源

Three runtimes, one job each

来源

Keep inference off the main thread

来源

If the worker path fails (no WebGPU, a download error), it falls back to the old behavior instead of breaking the node.

来源