How to Trace Which Neurons Actually Caused Your LLM's Output, From the Command Line

Описание и применение

NeuronScope — это CLI‑инструмент для механистической интерпретируемости, позволяющий трассировать внимание и нейроны в открытых LLM, выдавая структурированный JSON, пригодный для CI и агентов.

Задача

Existing LLM explanations are second‑generation outputs from the model itself, offering no real insight into the internal components that produced a prediction. Researchers and developers need a way to trace the actual attention heads and MLP neurons responsible for a model’s output, especially for small models that can run locally.

Результат

NeuronScope enables users to trace which attention heads and MLP neurons drove a model’s prediction, producing structured JSON that can be consumed by CI pipelines or agents. It fills a gap in the ecosystem by providing a local, open‑weight, command‑line tool.

Рабочий процесс

• 1. Install NeuronScope via pip: pip install neuronscope-cli
• 2. Run a trace command: neuronscope trace gpt2 "The capital of France is Paris. The capital of Japan is" --top-k 3 --json
• 3. Inspect the JSON output, which lists top neurons, top heads, and model metadata
• 4. Optionally use the circuit command to chain ranking with single‑component ablation for a causal sketch

Доказательства

NeuronScope is our attempt at the alternative: a CLI and MCP server that traces which attention heads and MLP neurons actually drove a model's output, instead of asking the model to narrate itself.

Источник

It ships a versioned JSON schema on every command, and exposes the same four operations to an AI agent over MCP.

Источник