Running LLMs inside a Chrome extension: WebLLM, transformers.js and the MV3 traps

Описание и применение

В расширении Chrome AWFlow реализует локальные модели LLM, используя WebLLM, transformers.js и TensorFlow.js, выполняя инференс в отдельных Web Workers и поддерживая WebGPU, при отсутствии которого переключается на WASM.

Задача

Run AI nodes in a Chrome extension without an API key and without sending page data outside the machine.

Результат

The extension successfully runs local LLMs in the browser, keeping data local, providing a fallback to WASM for unsupported hardware, and maintaining UI responsiveness by isolating heavy inference in Web Workers.

Рабочий процесс

• 1. Ship three runtimes: WebLLM, transformers.js, TensorFlow.js. 2. Use dedicated Web Workers for each runtime to keep the UI thread responsive. 3. Route chat-style tasks to the WebLLM worker. 4. Route task pipelines (summarization, classification, embeddings) to a second worker shared by transformers.js and TensorFlow.js. 5. Detect WebGPU availability once; if present, use GPU for supported tasks, otherwise fall back to WASM. 6. Wrap WebLLM in a worker using LangChain’s ChatWebLLM, swapping in a worker engine when available. 7. If the worker path fails (no WebGPU, download error), fall back to the main‑thread engine without breaking the node.

Доказательства

I wanted that step to work without an API key and without sending the page anywhere, which meant running the model inside the extension itself.

Источник

Three runtimes, one job each

Источник

Keep inference off the main thread

Источник

If the worker path fails (no WebGPU, a download error), it falls back to the old behavior instead of breaking the node.

Источник