I Built a 188M Mixture-of-Experts LLM From Scratch on a Free GPU

Описание и применение

Автор создал 188‑млн‑параметрическую модель Mixture‑of‑Experts, активирующую ~51 млн параметров за токен, обученную на бесплатном GPU Tesla T4 за ~3 ч, демонстрируя, что небольшие MoE‑модели могут достигать мощности больших моделей при низкой вычислительной нагрузке.

Задача

Demonstrate that a small‑scale mixture‑of‑experts (MoE) language model can achieve large‑model capacity at the compute cost of a dense 51M model, all trained on a free Tesla T4 GPU for $0.

Результат

The DeepSeekMoE‑tiny model achieved 188,269,568 total parameters with only 51,430,400 active per token, training stably on a free T4 for ~2.5–3 hours, demonstrating that tiny MoE can match dense capacity at reduced compute.

Рабочий процесс

• Extract 12 MoE primitives and 11 meta‑patterns from literature
• Resolve TRIZ contradiction via segmentation, taking out, local quality
• Generate 5 candidate architectures with a morphological Zwicky box
• Score candidates on Function ÷ active‑params and run 300‑step CPU experiments
• Select DeepSeekMoE‑tiny based on routing balance and stability
• Train on TinyStories and mixed instruction data on free Colab T4
• Monitor guardrails (routing‑collapse, aux‑loss dominance, OOM, NaN, checkpoints)
• Publish architecture, model, and trainer code

Доказательства

I built a 188M Mixture‑of‑Experts LLM From Scratch on a Free GPU

Источник

188,269,568 total params, 51,430,400 active per token — the compute of a ~51M dense model, ~3.7× its capacity

Источник

Training is running right now on a free Colab T4: Phase 1 on TinyStories (~200M tokens) for fluent language, then Phase 2 on mixed instruction data (~15M tokens, including my Shivacon domain sets and Dolly-15k). Total ~215M tokens, ~2.5–3 hours, $0.

Источник