Running a 4B model in 6 GB and training 3B parameters on a laptop

Описание и применение

На ноутбуке с 6 ГБ видеопамяти удалось запустить и дообучить модель Qwen3‑4B‑Instruct‑2507, используя потоковую загрузку слоёв и оптимизатор Adafactor. Сервис потребляет 4.5 ГБ VRAM и обрабатывает 3.4 токена/сек, а обучение 3.09 Млн параметров занимает 7.67 с/шаг при 4.93 ГБ VRAM.

Задача

Deploy and fine‑tune a 4‑B parameter LLM on a laptop GPU with only 6 GB VRAM.

Результат

Serving achieved 4.5 GB peak VRAM and 3.4 tok/s decode; RAG returned 50/50 retrieval and citation accuracy; training updated all 3.09 B parameters with 7.67 s/step and 4.93 GB peak VRAM, but only a 150‑step demo was run.

Рабочий процесс

• 1. Install LCGA Local and download Qwen3‑4B‑Instruct‑2507 weights.
• 2. Run "python llm.py" to start the server; weights are streamed and only one bf16 layer slot is resident.
• 3. For RAG, use the CPU‑only BM25 over the 326‑document Wikipedia corpus; the system returns passages and citations.
• 4. To train, execute the M5 experiment script which streams blocks and uses Adafactor to update all 3.09 B parameters.
• 5. Monitor VRAM usage and training metrics (cross‑entropy, step time).

Доказательства

The serving path uses Qwen3-4B-Instruct-2507 in int8 with layer streaming and one bf16 layer slot.

Источник

Measured on the 6 GB RTX 3050 Laptop: 4,513 MB peak VRAM, ~3.4 tok/s decode, 18.8 s first run, 2.4 s warm start.

Источник

Using block streaming and Adafactor, M5 updates all 3.09B parameters on the 6 GB RTX 3050 Laptop: 150-step demo, Cross-entropy: 2.5034 → 2.3242, 7.67 s/step, 4.93 GB peak VRAM.

Источник

The sparse variant reached 16.56 PPL, while dense attention reached 12.46 PPL.

Источник