An 80GB H100, and only 3B of the model could keep up with conversation

说明与用途

在80 GB H100 GPU上运行35B参数MoE模型，每个令牌仅激活3B权重，路由准确率92.6%，推理时间306 ms。

问题

Determining whether an 80 GB H100 GPU can support conversational AI with a 35 B‑parameter model, balancing memory fit and response latency.

结果

MoE enabled a 35 B model to run on an 80 GB GPU with per‑token computation comparable to a 3 B model, achieving high routing accuracy and acceptable latency for conversational use.

工作流

• 1. Assumed 80 GB VRAM sufficient for 35 B model
• 2. Ran 35 B model and observed only 3 B active per token
• 3. Noted high latency and misclassifications with smaller models
• 4. Adopted MoE Qwen3.6‑35B‑A3B
• 5. Measured accuracy (92.6 %) and inference time (306 ms)
• 6. Compared to 4 B/8 B models (≈82 % accuracy)
• 7. Concluded MoE balances memory and latency for conversation

证据

I used to think that 80 GB of VRAM would be enough to chat with a large model. In reality, I could only effectively run a 3B-parameter model—and that was without voice.

来源

We adopted Qwen3.6-35B-A3B. Although the entire model has 35B weights, it only uses 3B weights when processing each word.

来源

In the judgment to distinguish what kind of request the user's utterance is, Qwen3.6-35B-A3B had an accuracy rate of 92.6%. Ordinary qwen3:8b and qwen3.5:4b had accuracy rates in the 82% range. The judgment time for 35B-A3B was 306ms.

来源