Running Qwen 3.8 Flash Next (125B) on a RTX 4090 – 100 T/s on a Desktop

Description and use

A single RTX 4090 achieves ~100 T/s on Qwen 3.8 Flash Next (125B) with 4‑bit quantisation, torch.compile, and flash‑attention, enabling local, low‑cost, real‑time inference.

Problem

Running a 125 B‑parameter LLM locally requires multi‑GPU clusters or expensive cloud APIs, limiting accessibility for small teams.

Outcome

A single RTX 4090 achieves ~100 T/s on Qwen 3.8 Flash Next (125B) with 4‑bit quantisation, torch.compile, and flash‑attention, enabling local, low‑cost, real‑time inference.

Workflow

• Install system prerequisites and create a Python 3.11 virtual environment
• Upgrade pip and install torch 2.3.0+cu123, torchvision 0.18.0+cu123, bitsandbytes 0.43.1, transformers, accelerate, huggingface_hub
• Clone the Strata repo and install the helper package
• Login to Hugging Face CLI and download the 4‑bit quantised Qwen model
• Run the inference script run_qwen.py with desired parameters
• Integrate the model into a FastAPI or Flask service for production use

Evidence

the new Qwen 3.8 Flash Next (125B) model can be run on a consumer‑grade RTX 4090 at an impressive 100 trillion tokens per second (T/s).

Source

By proving that a single RTX 4090 can handle it, the barrier to entry drops dramatically.

Source

The repo ships a set of scripts, quantisation tricks, and a torch.compile‑friendly pipeline that squeezes every ounce of performance out of the 24 GB VRAM card.

Source

Quantisation is the hero – 4‑bit with bitsandbytes cuts VRAM usage to ~20 GB while preserving >90 % of the original quality for most conversational tasks.

Source

Compilation matters – torch.compile (beta) together with the flash‑attention kernel gives us the 100 T/s claim.

Source