Magnitude vs Ollama: Hardware‑Tuned Kernels Cut Inference Latency in Half

Description and use

Magnitude’s JIT‑tuned kernels halve TTFT and double throughput on M2 Max, RTX 4090, and AMD 7950X compared to Ollama’s generic kernels.

Problem

Running a 70B model on heterogeneous hardware exposes Ollama’s pre‑compiled GGML kernels, which are optimized for generic x86/ARM profiles and ignore chip‑specific SIMD widths, cache hierarchies, and memory bandwidth. This leads to inconsistent time‑to‑first‑token (TTFT) and tokens‑per‑second across similar hardware, and KV‑cache thrashing can degrade throughput by up to 40% on long contexts.

Outcome

Magnitude’s hardware‑tuned kernels cut TTFT by roughly half and doubled throughput on all three platforms, achieving 1.8–2.3× speedup over Ollama’s generic kernels.

Workflow

• 1. Load the model with Magnitude, triggering a first‑run benchmark of matrix ops, attention, and memory patterns.
• 2. Compile optimized kernels for the detected hardware configuration.
• 3. Run inference benchmarks measuring cold‑start latency, TTFT at 2K/4K/8K contexts, and sustained throughput over 512‑token generation.
• 4. Compare results against Ollama’s generic kernels.
• 5. Observe TTFT and throughput improvements across M2 Max, RTX 4090, and AMD 7950X.
• 6. Attribute gains to hardware‑specific optimizations such as tensor‑core INT4 usage on the RTX 4090 and Metal shader compilation on Apple Silicon.

Evidence

Your M2 Max, RTX 4090, or AMD 7950X all run the same matrix multiplication routines, ignoring chip‑specific SIMD widths, cache hierarchies, and memory bandwidth characteristics.

Source

Magnitude takes a different path: on first model load, it benchmarks matrix ops, attention mechanisms, and memory transfer patterns directly on your silicon, then compiles optimized kernels for that exact configuration.

Source

Results: Magnitude Delivers 1.8–2.3x Speedup

Source

M2 Max | Ollama | 420ms | 18.2
M2 Max | Magnitude | 210ms | 38.7
RTX 4090 | Ollama | 180ms | 52.1
RTX 4090 | Magnitude | 85ms | 119.3
7950X (CPU) | Ollama | 1840ms | 6.8
7950X (CPU) | Magnitude | 950ms | 12.4

Source