Running RAG with a (small) language model in the browser

说明与用途

在浏览器中运行的小型语言模型在检索增强生成（RAG）中往往产生不可靠的答案。作者构建了完整的客户端系统，评估了几种 300‑350 MB 的模型，发现它们的可信度仅约 10%。引入了相关性门控和重复惩罚，但可信度问题仍然存在。

问题

Small language models run in the browser often produce unfaithful or nonsensical answers when used for retrieval‑augmented generation (RAG).

结果

None of the evaluated models under 350 MB produced faithful answers more than 10 % of the time; they often hallucinated or repeated tokens. A relevance gate and repetition penalty reduced obvious hallucinations but did not solve faithfulness. The 579 MB Qwen3‑0.6B was the most reliable but still only faithful half the time.

工作流

• 1. User enters a question. 2. Static‑embedding search retrieves four top passages. 3. If the best passage score < threshold, abort and inform user. 4. Construct prompt with passages and system instruction. 5. Run chosen small model in a Web Worker via transformers.js (WebGPU or WASM). 6. Stream two‑sentence answer to the page.
• 7. Provide buttons to load model, clear cache, or switch models.

证据

"Turns out nothing off the shelf under 350 MB is usable for this. They aren't slightly worse; they're faithful at most three times out of thirty"

来源

"SmolLM2-360M is 390 MB as q4 and 275 MB as q4f16, with identical output. Everything below uses q4f16."

来源

"If the best passage scores below a threshold, the model isn't called at all and the page says it couldn't find this on the blog."

来源