How to Trace Which Neurons Actually Caused Your LLM's Output, From the Command Line

Description and use

NeuronScope is a CLI tool for mechanistic interpretability that traces attention heads and neurons in open‑weight LLMs, outputting structured JSON suitable for CI pipelines and agent workflows.

Problem

Existing LLM explanations are second‑generation outputs from the model itself, offering no real insight into the internal components that produced a prediction. Researchers and developers need a way to trace the actual attention heads and MLP neurons responsible for a model’s output, especially for small models that can run locally.

Outcome

NeuronScope enables users to trace which attention heads and MLP neurons drove a model’s prediction, producing structured JSON that can be consumed by CI pipelines or agents. It fills a gap in the ecosystem by providing a local, open‑weight, command‑line tool.

Workflow

• 1. Install NeuronScope via pip: pip install neuronscope-cli
• 2. Run a trace command: neuronscope trace gpt2 "The capital of France is Paris. The capital of Japan is" --top-k 3 --json
• 3. Inspect the JSON output, which lists top neurons, top heads, and model metadata
• 4. Optionally use the circuit command to chain ranking with single‑component ablation for a causal sketch

Evidence

NeuronScope is our attempt at the alternative: a CLI and MCP server that traces which attention heads and MLP neurons actually drove a model's output, instead of asking the model to narrate itself.

Source

It ships a versioned JSON schema on every command, and exposes the same four operations to an AI agent over MCP.

Source