I Built a 188M Mixture-of-Experts LLM From Scratch on a Free GPU

说明与用途

作者在免费Tesla T4 GPU上训练了一个188M参数的Mixture‑of‑Experts LLM，每个token仅激活约51M参数，耗时约3小时，证明小规模MoE可在低计算成本下实现大模型容量。

问题

Demonstrate that a small‑scale mixture‑of‑experts (MoE) language model can achieve large‑model capacity at the compute cost of a dense 51M model, all trained on a free Tesla T4 GPU for $0.

结果

The DeepSeekMoE‑tiny model achieved 188,269,568 total parameters with only 51,430,400 active per token, training stably on a free T4 for ~2.5–3 hours, demonstrating that tiny MoE can match dense capacity at reduced compute.

工作流

• Extract 12 MoE primitives and 11 meta‑patterns from literature
• Resolve TRIZ contradiction via segmentation, taking out, local quality
• Generate 5 candidate architectures with a morphological Zwicky box
• Score candidates on Function ÷ active‑params and run 300‑step CPU experiments
• Select DeepSeekMoE‑tiny based on routing balance and stability
• Train on TinyStories and mixed instruction data on free Colab T4
• Monitor guardrails (routing‑collapse, aux‑loss dominance, OOM, NaN, checkpoints)
• Publish architecture, model, and trainer code

证据

I built a 188M Mixture‑of‑Experts LLM From Scratch on a Free GPU

来源

188,269,568 total params, 51,430,400 active per token — the compute of a ~51M dense model, ~3.7× its capacity

来源

Training is running right now on a free Colab T4: Phase 1 on TinyStories (~200M tokens) for fluent language, then Phase 2 on mixed instruction data (~15M tokens, including my Shivacon domain sets and Dolly-15k). Total ~215M tokens, ~2.5–3 hours, $0.

来源