The Flag We Tuned Around Got Deleted

Description and use

Deleting the -sm row flag in llama.cpp caused a throughput regression, but parallel slots and speculative decoding restored and surpassed the former performance.

Problem

A critical performance flag (-sm row) for multi‑GPU llama.cpp inference was deleted upstream, causing a regression in throughput and forcing a re‑tuning of the stack.

Outcome

The deletion of the row split flag forced a shift to layer split, but throughput was recovered and surpassed via parallel slots and speculative decoding, demonstrating that performance gains can come from orthogonal features.

Workflow

• 1. Build model with row split on dual GPUs. 2. Measure 12‑14 tokens/sec generation. 3. In July 2026, upstream deletes -sm row. 4. Rebuild with layer split; measure 8.46 tokens/sec. 5. Enable --parallel 4; measure 12.8/15.0 tokens/sec. 6. Enable MTP speculative decoding; measure 13.3 tokens/sec.
• 7. Verify output correctness and acceptance rates.

Evidence

The single most important llama.cpp flag for my dual Tesla P40 setup was -sm row. It split every layer's tensors across both GPUs and it was worth nearly double the throughput of the alternative: 12-14 tokens/sec against about 7 for layer split.

Source

In July 2026, upstream llama.cpp deleted it. Not deprecated. Deleted.

Source

On the new stack, layer split alone measured 8.46 tokens/sec single-stream, right where the old "layer is about 7" rule predicted.

Source

--parallel 4 (four concurrent slots): 8.46 / 12.8 / 15.0 tokens/sec at 1, 2, and 4 slots. Aggregate throughput nearly doubles before per-slot latency degrades.

Source

MTP speculative decoding: 8.46 to about 13.3 tokens/sec single-stream, a 57% lift, acceptance rates 0.38 to 0.63, output correctness verified.

Source