Running RAG with a (small) language model in the browser

Описание и применение

Маленькие языковые модели, работающие в браузере, часто генерируют недостоверные ответы при использовании в RAG. Автор создал полностью клиентскую систему, оценил несколько моделей до 350 МБ и обнаружил, что они дают достоверные ответы лишь в 10 % случаев. Введены пороговый фильтр релевантности и штраф за повторения, но проблемы с достоверностью остаются.

Задача

Small language models run in the browser often produce unfaithful or nonsensical answers when used for retrieval‑augmented generation (RAG).

Результат

None of the evaluated models under 350 MB produced faithful answers more than 10 % of the time; they often hallucinated or repeated tokens. A relevance gate and repetition penalty reduced obvious hallucinations but did not solve faithfulness. The 579 MB Qwen3‑0.6B was the most reliable but still only faithful half the time.

Рабочий процесс

• 1. User enters a question. 2. Static‑embedding search retrieves four top passages. 3. If the best passage score < threshold, abort and inform user. 4. Construct prompt with passages and system instruction. 5. Run chosen small model in a Web Worker via transformers.js (WebGPU or WASM). 6. Stream two‑sentence answer to the page.
• 7. Provide buttons to load model, clear cache, or switch models.

Доказательства

"Turns out nothing off the shelf under 350 MB is usable for this. They aren't slightly worse; they're faithful at most three times out of thirty"

Источник

"SmolLM2-360M is 390 MB as q4 and 275 MB as q4f16, with identical output. Everything below uses q4f16."

Источник

"If the best passage scores below a threshold, the model isn't called at all and the page says it couldn't find this on the blog."

Источник