I Built a 188M Mixture-of-Experts LLM From Scratch on a Free GPU

Description and use

The author built a 188M‑parameter MoE LLM that activates ~51M parameters per token, trained on a free Tesla T4 GPU in ~3 hours, proving tiny MoE can match large‑model capacity at reduced compute.

Problem

Demonstrate that a small‑scale mixture‑of‑experts (MoE) language model can achieve large‑model capacity at the compute cost of a dense 51M model, all trained on a free Tesla T4 GPU for $0.

Outcome

The DeepSeekMoE‑tiny model achieved 188,269,568 total parameters with only 51,430,400 active per token, training stably on a free T4 for ~2.5–3 hours, demonstrating that tiny MoE can match dense capacity at reduced compute.

Workflow

• Extract 12 MoE primitives and 11 meta‑patterns from literature
• Resolve TRIZ contradiction via segmentation, taking out, local quality
• Generate 5 candidate architectures with a morphological Zwicky box
• Score candidates on Function ÷ active‑params and run 300‑step CPU experiments
• Select DeepSeekMoE‑tiny based on routing balance and stability
• Train on TinyStories and mixed instruction data on free Colab T4
• Monitor guardrails (routing‑collapse, aux‑loss dominance, OOM, NaN, checkpoints)
• Publish architecture, model, and trainer code

Evidence

I built a 188M Mixture‑of‑Experts LLM From Scratch on a Free GPU

Source

188,269,568 total params, 51,430,400 active per token — the compute of a ~51M dense model, ~3.7× its capacity

Source

Training is running right now on a free Colab T4: Phase 1 on TinyStories (~200M tokens) for fluent language, then Phase 2 on mixed instruction data (~15M tokens, including my Shivacon domain sets and Dolly-15k). Total ~215M tokens, ~2.5–3 hours, $0.

Source