Running LLMs inside a Chrome extension: WebLLM, transformers.js and the MV3 traps

Description and use

AWFlow Chrome extension runs local LLMs using WebLLM, transformers.js, and TensorFlow.js, performing inference in dedicated Web Workers and supporting WebGPU with a WASM fallback.

Problem

Run AI nodes in a Chrome extension without an API key and without sending page data outside the machine.

Outcome

The extension successfully runs local LLMs in the browser, keeping data local, providing a fallback to WASM for unsupported hardware, and maintaining UI responsiveness by isolating heavy inference in Web Workers.

Workflow

• 1. Ship three runtimes: WebLLM, transformers.js, TensorFlow.js. 2. Use dedicated Web Workers for each runtime to keep the UI thread responsive. 3. Route chat-style tasks to the WebLLM worker. 4. Route task pipelines (summarization, classification, embeddings) to a second worker shared by transformers.js and TensorFlow.js. 5. Detect WebGPU availability once; if present, use GPU for supported tasks, otherwise fall back to WASM. 6. Wrap WebLLM in a worker using LangChain’s ChatWebLLM, swapping in a worker engine when available. 7. If the worker path fails (no WebGPU, download error), fall back to the main‑thread engine without breaking the node.

Evidence

I wanted that step to work without an API key and without sending the page anywhere, which meant running the model inside the extension itself.

Source

Three runtimes, one job each

Source

Keep inference off the main thread

Source

If the worker path fails (no WebGPU, a download error), it falls back to the old behavior instead of breaking the node.

Source