Running a 4B model in 6 GB and training 3B parameters on a laptop

Description and use

On a laptop with 6 GB VRAM, the Qwen3‑4B‑Instruct‑2507 model was served using int8 layer streaming, achieving 4.5 GB peak VRAM and 3.4 tok/s. Full‑parameter training of 3.09 B parameters was performed with block streaming and Adafactor, using 4.93 GB VRAM and 7.67 s/step.

Problem

Deploy and fine‑tune a 4‑B parameter LLM on a laptop GPU with only 6 GB VRAM.

Outcome

Serving achieved 4.5 GB peak VRAM and 3.4 tok/s decode; RAG returned 50/50 retrieval and citation accuracy; training updated all 3.09 B parameters with 7.67 s/step and 4.93 GB peak VRAM, but only a 150‑step demo was run.

Workflow

• 1. Install LCGA Local and download Qwen3‑4B‑Instruct‑2507 weights.
• 2. Run "python llm.py" to start the server; weights are streamed and only one bf16 layer slot is resident.
• 3. For RAG, use the CPU‑only BM25 over the 326‑document Wikipedia corpus; the system returns passages and citations.
• 4. To train, execute the M5 experiment script which streams blocks and uses Adafactor to update all 3.09 B parameters.
• 5. Monitor VRAM usage and training metrics (cross‑entropy, step time).

Evidence

The serving path uses Qwen3-4B-Instruct-2507 in int8 with layer streaming and one bf16 layer slot.

Source

Measured on the 6 GB RTX 3050 Laptop: 4,513 MB peak VRAM, ~3.4 tok/s decode, 18.8 s first run, 2.4 s warm start.

Source

Using block streaming and Adafactor, M5 updates all 3.09B parameters on the 6 GB RTX 3050 Laptop: 150-step demo, Cross-entropy: 2.5034 → 2.3242, 7.67 s/step, 4.93 GB peak VRAM.

Source

The sparse variant reached 16.56 PPL, while dense attention reached 12.46 PPL.

Source