Running a 4B model in 6 GB and training 3B parameters on a laptop

说明与用途

在6 GB显存的笔记本上，使用int8层流式加载和Adafactor优化器成功部署并训练了Qwen3‑4B‑Instruct‑2507模型。服务峰值显存为4.5 GB，解码速度约3.4 tok/s；完整参数训练使用4.93 GB显存，步长时间为7.67 s。

问题

Deploy and fine‑tune a 4‑B parameter LLM on a laptop GPU with only 6 GB VRAM.

结果

Serving achieved 4.5 GB peak VRAM and 3.4 tok/s decode; RAG returned 50/50 retrieval and citation accuracy; training updated all 3.09 B parameters with 7.67 s/step and 4.93 GB peak VRAM, but only a 150‑step demo was run.

工作流

• 1. Install LCGA Local and download Qwen3‑4B‑Instruct‑2507 weights.
• 2. Run "python llm.py" to start the server; weights are streamed and only one bf16 layer slot is resident.
• 3. For RAG, use the CPU‑only BM25 over the 326‑document Wikipedia corpus; the system returns passages and citations.
• 4. To train, execute the M5 experiment script which streams blocks and uses Adafactor to update all 3.09 B parameters.
• 5. Monitor VRAM usage and training metrics (cross‑entropy, step time).

证据

The serving path uses Qwen3-4B-Instruct-2507 in int8 with layer streaming and one bf16 layer slot.

来源

Measured on the 6 GB RTX 3050 Laptop: 4,513 MB peak VRAM, ~3.4 tok/s decode, 18.8 s first run, 2.4 s warm start.

来源

Using block streaming and Adafactor, M5 updates all 3.09B parameters on the 6 GB RTX 3050 Laptop: 150-step demo, Cross-entropy: 2.5034 → 2.3242, 7.67 s/step, 4.93 GB peak VRAM.

来源

The sparse variant reached 16.56 PPL, while dense attention reached 12.46 PPL.

来源