How to Trace Which Neurons Actually Caused Your LLM's Output, From the Command Line

说明与用途

NeuronScope 是一个用于机制可解释性的命令行工具，可追踪开放权重 LLM 中的注意力头和神经元，并输出结构化 JSON，适用于 CI 和代理工作流。

问题

Existing LLM explanations are second‑generation outputs from the model itself, offering no real insight into the internal components that produced a prediction. Researchers and developers need a way to trace the actual attention heads and MLP neurons responsible for a model’s output, especially for small models that can run locally.

结果

NeuronScope enables users to trace which attention heads and MLP neurons drove a model’s prediction, producing structured JSON that can be consumed by CI pipelines or agents. It fills a gap in the ecosystem by providing a local, open‑weight, command‑line tool.

工作流

• 1. Install NeuronScope via pip: pip install neuronscope-cli
• 2. Run a trace command: neuronscope trace gpt2 "The capital of France is Paris. The capital of Japan is" --top-k 3 --json
• 3. Inspect the JSON output, which lists top neurons, top heads, and model metadata
• 4. Optionally use the circuit command to chain ranking with single‑component ablation for a causal sketch

证据

NeuronScope is our attempt at the alternative: a CLI and MCP server that traces which attention heads and MLP neurons actually drove a model's output, instead of asking the model to narrate itself.

来源

It ships a versioned JSON schema on every command, and exposes the same four operations to an AI agent over MCP.

来源