Magnitude vs Ollama: Hardware‑Tuned Kernels Cut Inference Latency in Half

Описание и применение

Magnitude использует JIT‑тюнинг ядра, чтобы адаптировать LLM‑инференс к конкретному железу, сокращая время первого токена и удваивая пропускную способность по сравнению с Ollama.

Задача

Running a 70B model on heterogeneous hardware exposes Ollama’s pre‑compiled GGML kernels, which are optimized for generic x86/ARM profiles and ignore chip‑specific SIMD widths, cache hierarchies, and memory bandwidth. This leads to inconsistent time‑to‑first‑token (TTFT) and tokens‑per‑second across similar hardware, and KV‑cache thrashing can degrade throughput by up to 40% on long contexts.

Результат

Magnitude’s hardware‑tuned kernels cut TTFT by roughly half and doubled throughput on all three platforms, achieving 1.8–2.3× speedup over Ollama’s generic kernels.

Рабочий процесс

• 1. Load the model with Magnitude, triggering a first‑run benchmark of matrix ops, attention, and memory patterns.
• 2. Compile optimized kernels for the detected hardware configuration.
• 3. Run inference benchmarks measuring cold‑start latency, TTFT at 2K/4K/8K contexts, and sustained throughput over 512‑token generation.
• 4. Compare results against Ollama’s generic kernels.
• 5. Observe TTFT and throughput improvements across M2 Max, RTX 4090, and AMD 7950X.
• 6. Attribute gains to hardware‑specific optimizations such as tensor‑core INT4 usage on the RTX 4090 and Metal shader compilation on Apple Silicon.

Доказательства

Your M2 Max, RTX 4090, or AMD 7950X all run the same matrix multiplication routines, ignoring chip‑specific SIMD widths, cache hierarchies, and memory bandwidth characteristics.

Источник

Magnitude takes a different path: on first model load, it benchmarks matrix ops, attention mechanisms, and memory transfer patterns directly on your silicon, then compiles optimized kernels for that exact configuration.

Источник

Results: Magnitude Delivers 1.8–2.3x Speedup

Источник

M2 Max | Ollama | 420ms | 18.2
M2 Max | Magnitude | 210ms | 38.7
RTX 4090 | Ollama | 180ms | 52.1
RTX 4090 | Magnitude | 85ms | 119.3
7950X (CPU) | Ollama | 1840ms | 6.8
7950X (CPU) | Magnitude | 950ms | 12.4

Источник