Skip to content

Mechanistic Interpretability

Interpretability tells you which parts of an input mattered for a prediction. Mechanistic interpretability asks how the network computes it:

  • what its layers represent;
  • which directions in its activations stand for concepts;
  • which of its weights do the work;
  • how information flows from the prompt to the answer.

This section covers the tools pulsatrix has for that, from reading activations to full attribution graphs of a language model's prediction.

Which tool should I use?

Your question Tool Page
Is concept X linearly readable from layer L? LinearProbe, TrainLinearProbe(), with MeasureSelectivity() or CompareToBaseline() as the control Probes and concepts
How similar are two layers, two models, or a model before and after fine-tuning? LinearCka, LinearCkaMatrix Representations
Does a model order the inputs as a brain recording or a model of the categories does? Is the agreement chance? RepresentationalDissimilarity, CompareRdms, RsaPermutationTest (RSA) Representations
Which directions do two layers share? Reproduce a result reported with SVCCA. Svcca, CanonicalCorrelations Representations
Which inputs does a layer treat as alike? Do two models agree on it? NearestNeighbors, MutualKnnAlignment, MutualNearestNeighbors, MeasureHubness Representations
Is cosine similarity meaningful in this layer? How many dimensions does it use? Anisotropy, MeasureEffectiveRank, TwoNNDimension Representations
What does a layer look like in 2D or 3D? Which directions vary most? FitPca, PcaProject, PcaSplitHalfOverlap, RenderProjectionSvg Representations
How do I look at a layer with UMAP or t-SNE? ExportEmbeddings, WriteNpy, ReadNpy, MakeProjectionDocument(coordinates, "umap") Representations
How do I turn a layer's activations into points for topological data analysis? PointCloud, MakePointCloud Topology
What are the distances between those points, and how far off can float32 make them? PairwiseDistances, DistanceMatrix::error_bound Topology
Which clusters does a layer form, and over what range of scales do they stay apart? ComputeH0, SingleLinkageClusters Topology
Does a layer arrange inputs in loops or around voids? Which inputs form them? ComputeRipsPersistence, critical edges, cocycles Topology
Do two layers, models or seeds have the same topology, within the floating-point error? BottleneckDistance (certified), WassersteinDistance Topology
How do I feed a diagram to a probe, a test or a plot? ComputePersistenceLandscape (certified), BettiCurve, ComputePersistenceImage, PersistentEntropy Topology
How do I run persistent homology on 100,000 activations? FarthestPointLandmarks, WithSubsamplingBound Topology
Is a loop real, or noise? UniversalNullTest, BootstrapBand, GaussianBaseline, BaselinePValue Topology
Do two layers have the same clusters and loops, which CKA can miss? Rtd, CompareRepresentations (CKA, mutual kNN and RTD together) Topology
Does a concept, given by examples, push the model towards a class? RunTcav, TrainCav Probes and concepts
Which of a layer's channels does an LRP explanation pass through, and where in the input is each? ChannelRelevance, ConditionalHeatmaps, ConceptReferenceSamples (CRP) LRP
Which directions in a layer stand for concepts, found without labels? Sparse autoencoders: TopKSparseAutoencoder, JumpReLUSparseAutoencoder, SparseAutoencoder Finding features
Which concepts take more than one dimension, such as an angle or a position? BlockSparseFeaturizer Finding features
What does an MLP compute, step by step? Transcoder Finding features
Are my features any good? EvaluateReconstruction, MeasureLossRecovered, FeatureAbsorption, with baselines Evaluating features
Can a direction push the model's behavior, and on how many inputs does it work? SteeringHook, MeasureSteering Steering
What did fine-tuning change in a model? Crosscoder, MeasureLatentScaling, pulsatrix_diff_lm Model diffing
Which pieces of the weights does each input use? ParameterDecomposition (SPD, VPD) Parameter decomposition
Which features carry a language model's prediction, layer to layer? TraceCircuit, pulsatrix_explain_text --transcoders Attribution graphs
Which layers does a small network's output depend on? ExplainerContext::build_circuit_graph() Attribution graphs
What happens if I overwrite one activation? ExplainerContext::forward_pass_with_patch() Below
What would the model predict if layer L were its last? ExplainerContext::logit_lens() Below
How do I sample outcomes in proportion to a reward? GFlowNets GFlowNets

A typical workflow

For a researcher starting on a model, the pieces fit together like this:

  1. Collect activations from the layer you care about, with a hook (below).
  2. Describe the layer: how many dimensions it uses, whether cosine similarity means anything in it, and how it compares with other layers and models (Representations).
  3. Probe for the concepts you already have labels for, with a control. A probe tells you whether the information is there (Probes and concepts).
  4. Train a featurizer (a TopK or BatchTopK SAE is a good default) to find the directions the model uses without being told what to look for.
  5. Evaluate it against baselines: the same featurizer on a randomly initialized copy of the model, and a linear probe. A feature that also shows up in a random model says nothing about what the trained one learned.
  6. Test causally.
  7. Splice the reconstructions back in and see how much of the loss you keep (loss recovered).
  8. Steer along a direction, with a random-direction control.
  9. For one prediction, trace an attribution graph.

Each page reports what these steps found on real models (ESM-2 protein models, SmolLM2, Gemma 3), including results that didn't work out. Read those before relying on a method.

Building blocks

Reading and changing activations

Language models expose two hooks, used throughout this section:

  • HiddenStateHook on CausalLM and EncoderLM. It is called with each residual-stream position during forward(): 0 for the embeddings, i for the output of block i. It returns the tensor the model continues with. Return it unchanged to only read.
  • MlpHook on each TransformerBlock and EncoderBlock. It sees an MLP's input (after the block's norm) and its output, and returns what is added to the residual stream.
#include "pulsatrix/causal_lm.hpp"
#include "pulsatrix/cpu_backend.hpp"
#include "pulsatrix/tokenizer_json.hpp"

using namespace pulsatrix;

CPUBackend backend;
std::unique_ptr<CausalLM> model = LoadCausalLM("models/SmolLM2-135M", &backend);
const TextTokenizer tok = LoadTokenizerJson("models/SmolLM2-135M/tokenizer.json");
const int64_t d = model->config().hidden_size;

// corpus: your texts, a std::vector<std::string>.
// Collect the residual stream after block 6, one row per token.
std::vector<float> rows;
model->set_hidden_state_hook([&](int64_t position, const Tensor& h) {
    if (position == 6) {
        const std::vector<float> v = h.to_host_vector();   // (1, tokens, d)
        rows.insert(rows.end(), v.begin(), v.end());
    }
    return h;                                              // unchanged: only read
});
for (const std::string& text : corpus) {
    const std::vector<int64_t> ids = tok.encode(text).ids;
    const std::vector<float> ids_f(ids.begin(), ids.end());
    (void)model->forward(Tensor(Shape({1, static_cast<int64_t>(ids.size())}), &backend, ids_f));
}
model->set_hidden_state_hook({});
const Tensor activations(Shape({static_cast<int64_t>(rows.size()) / d, d}), &backend, rows);

For small networks built from layers, ExplainerContext caches every layer's output and offers the classic tools: - activation_snapshot(): a copy of one forward pass's activations that stays valid after later passes. - forward_pass_with_patch(): activation patching, overwriting one activation. - logit_lens(): what the model would predict if a given layer were the last. - attention_weights().

Baselines

NullModelBaseline re-initializes a model randomly, so the same probe, featurizer or explanation can run on it (Interpretability). It is the single most useful check in this section. In the ESM-2 results on the following pages, a random model's SAE explains more variance than the trained model's, and some "concept features" appear in it too.

Command-line tools

Tool What it does Page
pulsatrix_probe_esm Probes, SAEs, transcoders and block-sparse featurizers on an ESM-2 layer, with concept matching, absorption, loss recovered and random-model baselines Finding features, Protein language models
pulsatrix_steer_esm A steering vector on ESM-2, with the reliability report Steering
pulsatrix_diff_lm A crosscoder between a base language model and its fine-tune, with Latent Scaling Model diffing
pulsatrix_spd_toy Parameter decomposition (SPD, VPD) of a toy model of superposition Parameter decomposition
pulsatrix_explain_text --graph An attribution graph for a prompt, from AttnLRP or (with --transcoders) transcoder features, for Neuronpedia's viewer Attribution graphs

Each prints its usage with --help. Each source file (in tools/) starts with a comment that explains every option.

Recipes

Full API reference: Doxygen: Mechanistic Interpretability