My Q4 model said True where BF16 said False, so I diffed them layer by layer

Description and use

heterodyne diff two llama.cpp runs token‑by‑token and layer‑by‑layer, revealing that Q4_K_M vs BF16 divergence originates gradually across all layers, with attention and MLP blocks contributing roughly equally; KV cache and batch size also affect the drift.

Problem

Quantized GGUF models (Q4_K_M) produce different predictions than BF16 models, but aggregate metrics hide where the divergence originates.

Outcome

heterodyne revealed that Q4_K_M vs BF16 divergence originates gradually across all 28 layers, with attention and MLP blocks contributing roughly equally; restoring BF16 residuals in Q4_K_M reduces KL from 0.48 to 0.04 between layers 6–12. Quantization of KV cache and batch size also significantly affect divergence.

Workflow

• 1. Define two arms: each arm is a GGUF file plus runtime settings (e.g., cache type, flash attention, batch size).
• 2. Run both arms on identical tokens in llama.cpp, recording per‑position KL(A‖B), top‑1 changes, and residual streams after every layer.
• 3. Compute a numerical floor by re‑running arm A with an odd batch split to gauge harmless differences.
• 4. Perform transplants: write arm A’s residual (l_out‑N) into arm B and measure KL(A‖patched B), and vice versa, for each layer.
• 5. Separate attention and MLP contributions by restoring arm A’s residual at ffn_inp‑N and l_out‑N and computing KL differences between steps.
• 6. Aggregate results to identify which layers and blocks carry divergence to the output.
• 7. Verify with known tests (e.g., K‑cache only differences should affect only attention).
• 8. Optionally run additional configurations (different quantizations, cache types, batch sizes) to attribute changes.
• 9. Store experiment files with provenance (GGUF hashes, llama.cpp commit, build flags) and verify reproducibility with heterodyne verify.

Evidence

Qwen3-0.6B in BF16 reads this: … and puts 97.9% on False. The Q4_K_M quant of the same model puts 79.1% on True.

Source

heterodyne runs two configurations of llama.cpp (two arms: a GGUF file plus the runtime settings that can change its numbers) on identical tokens and shows…

Source

The 12-chunk run took 29 minutes on a laptop i7-1355U.

Source

All on Qwen3-0.6B, same text, same laptop… The KV cache matters more than you'd think.

Source