Running RAG with a (small) language model in the browser

Description and use

Small language models running in the browser often produce unfaithful answers in RAG. The author built a client‑side system, evaluated several 300‑350 MB models, and found they were faithful only ~10 % of the time. A relevance gate and repetition penalty were added, but faithfulness issues persist.

Problem

Small language models run in the browser often produce unfaithful or nonsensical answers when used for retrieval‑augmented generation (RAG).

Outcome

None of the evaluated models under 350 MB produced faithful answers more than 10 % of the time; they often hallucinated or repeated tokens. A relevance gate and repetition penalty reduced obvious hallucinations but did not solve faithfulness. The 579 MB Qwen3‑0.6B was the most reliable but still only faithful half the time.

Workflow

• 1. User enters a question. 2. Static‑embedding search retrieves four top passages. 3. If the best passage score < threshold, abort and inform user. 4. Construct prompt with passages and system instruction. 5. Run chosen small model in a Web Worker via transformers.js (WebGPU or WASM). 6. Stream two‑sentence answer to the page.
• 7. Provide buttons to load model, clear cache, or switch models.

Evidence

"Turns out nothing off the shelf under 350 MB is usable for this. They aren't slightly worse; they're faithful at most three times out of thirty"

Source

"SmolLM2-360M is 390 MB as q4 and 275 MB as q4f16, with identical output. Everything below uses q4f16."

Source

"If the best passage scores below a threshold, the model isn't called at all and the page says it couldn't find this on the blog."

Source