Making Qwen Faster on an RTX 3090

Описание и применение

Ускорение генерации модели Qwen3.8‑27B на RTX 3090 с помощью IQ3_S‑весов, Q8‑KV‑кэша и MTP4 повысило скорость до ~60 токенов/сек, сохранив почти те же показатели прохождения задач.

Задача

Improve generation speed of Qwen3.8-27B on a single 24 GB RTX 3090 while maintaining functional and delivered pass rates for coding repair tasks.

Результат

Using IQ3_S weights, Q8 KV cache, and MTP4 on a single RTX 3090 increased generation speed from ~33 tokens/s to ~60 tokens/s (≈80 % faster) while maintaining comparable delivered pass rates (21/24). Ledger task failures were traced to output truncation and test interface defects, which were corrected to achieve full pass rates.

Рабочий процесс

• 1. Select Qwen3.8-27B model and quantization bundle (IQ3_S with MTP head).
• 2. Configure KV cache to Q8 format and set total context window to 128 K tokens.
• 3. Enable MTP4 (max 4 draft tokens) and set output cap to 64 K tokens.
• 4. Run eight repair tasks (six Python, two TypeScript) each once, measuring generation rate and delivered pass counts.
• 5. Compare against baseline (Q4_K_M, MTP off) and alternative (Bartowski IQ4_XS).
• 6. Adjust output cap for Ledger task to 64 K to resolve truncation.
• 7. Validate functional and delivered passes, noting any test defects.
• 8. Record GPU memory usage and task times.
• 9. Optionally test larger 220 K context window for long-input migration task.
• 10. Analyze trade-offs between speed, memory, and correctness.

Доказательства

Generation speed during coding tasks increased from roughly 33 tokens/s with the original configuration to around 60 tokens/s.

Источник

IQ3_S quantized weights, Q8 KV cache, and MTP4

Источник

The IQ3 bundle generated approximately 80% faster than the original Q4 bundle.

Источник

MTP4 was approximately 5.7% faster than MTP2; greater depth added no benefit.

Источник