An 80GB H100, and only 3B of the model could keep up with conversation

Description and use

An 80 GB H100 GPU can run a 35 B‑parameter MoE model, activating only 3 B of weights per token, achieving 92.6 % accuracy and 306 ms inference for routing.

Problem

Determining whether an 80 GB H100 GPU can support conversational AI with a 35 B‑parameter model, balancing memory fit and response latency.

Outcome

MoE enabled a 35 B model to run on an 80 GB GPU with per‑token computation comparable to a 3 B model, achieving high routing accuracy and acceptable latency for conversational use.

Workflow

• 1. Assumed 80 GB VRAM sufficient for 35 B model
• 2. Ran 35 B model and observed only 3 B active per token
• 3. Noted high latency and misclassifications with smaller models
• 4. Adopted MoE Qwen3.6‑35B‑A3B
• 5. Measured accuracy (92.6 %) and inference time (306 ms)
• 6. Compared to 4 B/8 B models (≈82 % accuracy)
• 7. Concluded MoE balances memory and latency for conversation

Evidence

I used to think that 80 GB of VRAM would be enough to chat with a large model. In reality, I could only effectively run a 3B-parameter model—and that was without voice.

Source

We adopted Qwen3.6-35B-A3B. Although the entire model has 35B weights, it only uses 3B weights when processing each word.

Source

In the judgment to distinguish what kind of request the user's utterance is, Qwen3.6-35B-A3B had an accuracy rate of 92.6%. Ordinary qwen3:8b and qwen3.5:4b had accuracy rates in the 82% range. The judgment time for 35B-A3B was 306ms.

Source