NeuronDump: look inside any local LLM and make it smaller

Описание и применение

NeuronDump позволяет пользователям локальных LLM анализировать и уменьшать размер GGUF‑файлов, удаляя ненужные слои, головы и токены, сохраняя при этом работоспособность модели.

Задача

Local LLM users download large GGUF files and treat them as black boxes, lacking insight into which model components are essential and which can be removed to reduce size without breaking functionality.

Результат

NeuronDump enables users to identify and remove non‑essential parts of GGUF models, producing smaller files that still load in llama.cpp or Ollama and maintain comparable performance.

Рабочий процесс

• 1. Install NeuronDump via prebuilt binaries or cargo.
• 2. Run neurondump list to discover local models.
• 3. Inspect a model with neurondump inspect <model> to view architecture, layers, heads, vocab, and template.
• 4. Use neurondump analyze <model> --out plan.json to generate a pruning plan.
• 5. Edit or review plan.json as needed.
• 6. Apply the plan with neurondump slim <model> --plan plan.json --only vocab --out small.gguf to create a trimmed GGUF.
• 7. (Optional) Build with --features trace to compare the cut model against the original in memory, evaluating token predictions, KL divergence, perplexity, and logit lens.

Доказательства

I wanted to open that box: see the layers, the attention heads, the vocabulary, the chat template, and then ask a simple question: which parts does this model not actually need, and what breaks if I remove them?

Источник

It reads GGUF files directly, with nothing hard-coded per model.

Источник

Cut each layer (or head) in turn, rank them by damage, and write the least important ones out as a smaller GGUF plus an Ollama Modelfile.

Источник