Analyzing Representations¶
These tools describe what a layer does with a set of inputs, without labels: how it compares with another layer or model, which inputs it treats as alike, how many dimensions it uses, and what it looks like in two or three dimensions.
Each takes a matrix with one row per input, (n, width), collected with the hooks described in
the overview. Two representations being compared
must come from the same n inputs, in the same order.
| Your question | Tool | Section |
|---|---|---|
| How similar are two layers, two models, or a model before and after fine-tuning? | LinearCka, LinearCkaMatrix |
CKA |
| Does a model order the inputs as a brain recording or a model of the categories does? | RsaPermutationTest |
RSA |
| Which directions do two layers share? | Svcca, CanonicalCorrelations |
SVCCA |
| Which inputs does a layer treat as alike? Do two models agree? | NearestNeighbors, MutualKnnAlignment |
Nearest neighbors |
| Is cosine similarity meaningful here? How many dimensions are used? | Anisotropy, MeasureEffectiveRank, TwoNNDimension |
The shape of a layer |
| What does a layer look like in 2D or 3D? | FitPca, RenderProjectionSvg |
PCA |
| How do I run UMAP or t-SNE on a layer? | ExportEmbeddings, ReadNpy |
UMAP and t-SNE |
For clusters, loops and voids, see Topology of representations. To compare two representations' topology next to CKA, use RTD and CompareRepresentations().
Compare against an untrained model. Every number on this page can come from the architecture
or the inputs rather than what was learned. Compute it on a
NullModelBaseline too.
Comparing representations (CKA)¶
Centered kernel alignment (CKA) measures how similar two representations of the same inputs are: one layer of two models, two layers of one model, a model before and after fine-tuning, or the same layer computed on the CPU and the GPU. Run the same n inputs through both, collect one row per input from each, and compare:
#include "pulsatrix/representation_similarity.hpp"
// layer_a: (n, 768) activations from one model; layer_b: (n, 1024) from another, same n inputs.
const double similarity = LinearCka(layer_a, layer_b);
- 1 means one representation is a rotation and uniform scaling of the other, plus an offset. The two layers don't need the same width, and their neurons don't need to line up.
- Near 0 means they are unrelated.
- Dimensions after the first are flattened, so a convolution's (n, channels, height, width) activations are n vectors. The tensors can be on any device; CKA is computed in double precision on the host.
LinearCkaMatrix(layers_a, layers_b) compares every layer of one model with every layer of
another, reusing each layer's work across its row or column. Draw the result as a heatmap:
#include "pulsatrix/viz/svg.hpp"
const CkaMatrix m = LinearCkaMatrix(layers_a, layers_b);
HeatmapDocument doc;
doc.title = "CKA, model A layers × model B layers";
doc.rows = m.rows;
doc.cols = m.cols;
doc.values.assign(m.values.begin(), m.values.end());
std::ofstream("cka.svg") << RenderHeatmapSvg(doc);
Things to know:
- Use the default, debiased estimator. The original estimator is pulled towards 1 when
there are few inputs for the width: two unrelated random representations of 50 inputs with
1,000 features each score 0.95 with it and −0.02 debiased. Pass
CkaEstimator::Biasedto reproduce numbers from papers that used it. The debiased value can be slightly negative, and needs at least 4 inputs. - Compare against an untrained model. Two layers can look similar because of the
architecture or the input, not what was learned. In the tests, the second hidden layers of two
small networks trained from different seeds score 0.68; one of them against the same network
untrained scores 0.10. Run the comparison on a
NullModelBaselinetoo (Baselines). - Outliers move it. A few inputs with very large activations can dominate, and CKA can be pushed up or down on purpose (Davari et al., 2022). Check that a result holds on a second set of inputs.
- A representation that is constant over the inputs has no CKA: the result is NaN.
The implementation matches the reference code of Kornblith et al. (ICML 2019) to 12 decimal places for both estimators.
RSA and SVCCA¶
Two older ways to compare representations are in the same header. Use CKA unless you need one of them.
RSA (representational similarity analysis) describes a representation by how far apart it puts every pair of inputs, its representational dissimilarity matrix (RDM), and correlates two RDMs. Because only the RDM is compared, the other side can be anything that orders the same inputs: another model, a brain recording, or a hand-made matrix that says which inputs share a category.
#include "pulsatrix/representation_similarity.hpp"
const Rdm model = RepresentationalDissimilarity(layer, RdmMetric::Correlation); // 1 − Pearson
const Rdm categories = ...; // e.g. 0 within a category, 1 across, in the same pair order
const RsaTestResult r = RsaPermutationTest(model, categories, RdmComparison::Spearman, 999);
// r.similarity: Spearman's correlation of the two RDMs; r.p_value: how often relabeled inputs
// agree as well.
- The metrics are correlation distance (the RSA default), Euclidean and cosine. RDMs are
stored as scipy's
pdiststores them, pairs (0, 1), (0, 2), ..., so they can go to or come from Python unchanged;Rdm::at(i, j)reads one. - Compare with Spearman (the default) when the two RDMs are on different scales, which is
almost always; only the order of the distances counts.
Rsa(x, y)does both steps. - Test with
RsaPermutationTest(), not a correlation's p-value. The distances in an RDM aren't independent (one input's row moves n − 1 of them), so the usual p-value is too small. The test relabels the inputs of one RDM instead. In the tests, two noisy views of four categories agree at 0.83 with p = 1/200, the smallest 199 relabelings can give, and unrelated representations reach p < 0.05 in 3 of 40 trials, about the one in 20 they should.
SVCCA keeps each representation's top singular directions, 99% of the variance by default, and finds the pairs of directions, one in each, that correlate most. The mean of those canonical correlations is the similarity.
const SvccaResult r = Svcca(layer_a, layer_b); // r.similarity, r.correlations (largest first)
const SvccaResult fixed = Svcca(layer_a, layer_b, SvccaOptions{0.99, 20}); // 20 directions each
Things to know:
- It is biased upwards, more than CKA. Canonical correlations are fitted to the inputs. Two unrelated 20-dimensional representations score 0.50 with 60 inputs, against −0.02 for debiased CKA, and still 0.09 with 2,000. Use many more inputs than kept directions.
- It refuses too few inputs. When the kept directions of both add up to the number of inputs
or more, some correlations are 1 whatever the data, and
Svcca()throws rather than report them. Ask for fewer directions withSvccaOptions::directions. - It ignores how a layer weights its directions. CCA is invariant to any invertible linear map: stretching one axis of a layer ten times and shearing another leaves SVCCA at 1, and drops CKA to 0.48 in the tests. Kornblith et al. found CKA better at matching corresponding layers for this reason.
CanonicalCorrelations(x, y)is plain CCA, on every direction with variance.
RSA matches independent numpy implementations of scipy's pdist and spearmanr to 12 decimal
places, and SVCCA and CCA one written with numpy's SVD and QR to 9.
Nearest neighbors¶
NearestNeighbors() finds, for each input, the k inputs a layer represents most alike, by cosine
similarity (the default) or Euclidean distance:
#include "pulsatrix/nearest_neighbors.hpp"
// embeddings: (n, width), a row per input.
NeighborOptions options;
options.k = 5;
options.center = true; // subtract the mean first; see below
const Neighbors near = NearestNeighbors(embeddings, options);
for (int64_t r = 0; r < near.k; ++r) {
std::cout << near.index(0, r) << " " << near.score(0, r) << "\n"; // input 0's neighbors, nearest first
}
- Search another set with
NearestNeighbors(queries, points, options), for example new inputs against a labelled reference set. - Match items across two sets with
MutualNearestNeighbors(a, b, options): the pairs (i, j) where each is among the other's k nearest. - Compare two models with
MutualKnnAlignment(a, b, options): for each input, the fraction of its k neighbors under one representation that are also its neighbors under the other, averaged. The two can have different widths. With the default options it is the mutual-kNN alignment of Huh et al.'s "Platonic Representation Hypothesis" (2024), and it matches their code exactly. Unrelated representations score about k / (n − 1) by chance, so read a score against that, not against 0. It complements CKA (above): CKA compares the whole geometry, mutual kNN only who is close to whom.
The search is exact, in double precision on the host. It costs about n² × width multiply-adds, so it suits up to tens of thousands of inputs. Ties keep the lower index first.
Things to know:
- Center cosine similarities. Activations often share a large common component, so every
pair's cosine is high and the ranking follows that component.
center = truesubtracts the points' mean first. It doesn't change Euclidean distances. -
Check for hubs. In high dimensions a few points become almost everyone's neighbor, and many become nobody's (Radovanović et al., 2010). Neighbor lists, and scores built on them, then say more about the hubs than about the inputs.
MeasureHubness(neighbors, n)reports how often each point is a neighbor, and the skewness of those counts: near 0 is even, well above 1 means hubs. For 300 random Gaussian points and k = 5:Dimension Skewness The biggest hub is in Never a neighbor 3 −0.04 3% of lists 2% of points 30 2.05 12% of lists 21% of points 300 3.52 20% of lists 36% of points Centering also reduces hubness for cosine similarity (Suzuki et al., 2013).
The shape of a layer¶
Three numbers describe a layer's representations of a set of inputs. Each is cheap enough to compute for every layer.
#include "pulsatrix/representation_geometry.hpp"
// layer: (n, width), a row per input.
const double raw = Anisotropy(layer); // mean cosine over all pairs
const double centered = Anisotropy(layer, /*center=*/true);
const EffectiveRank rank = MeasureEffectiveRank(layer); // .effective_rank, .participation_ratio
const TwoNNResult id = TwoNNDimension(layer); // .dimension, .mle_dimension
- Anisotropy is the mean cosine similarity between different inputs. Spread-out
directions score about 0. Language models' layers often score 0.5 or more (Ethayarajh,
2019): they share a few large directions, so any two inputs look alike by cosine. If raw
anisotropy is high and centered anisotropy is near 0, the similarity was the shared
direction. Center before comparing by cosine (
NeighborOptions::center, above). The test data shows this: rows spread evenly score 0.001, shifting them all along one axis raises that to 0.63, and centering brings it back to −0.003. - Effective rank (Roy and Vetterli, 2007) is how many directions the inputs spread along. It's the exponential of the entropy of the normalized singular values, between 1 and the smaller of n and the width. The participation ratio is a coarser count that the largest directions dominate more. Both center by default, because a shared direction would otherwise take up part of the count.
- TwoNN intrinsic dimension (Facco et al., 2017) is how many dimensions the inputs vary in
locally, which can be far below the width. It uses only each point's two nearest neighbors.
dimensionis the paper's fit and matches scikit-dimension and DADApy.mle_dimensionis DADApy's maximum-likelihood version. Exact repeated rows are dropped first and counted induplicates. In the test, points filling a 2- and a 5-dimensional cube, written in 40 dimensions, give 1.91 and 4.53.
Things to know about TwoNN:
- Low with too few points. It reads low when the dimension is high for the number of points: tens of dimensions need thousands.
- Noise raises it. It measures the nearest neighbors' scale, so noise on a low-dimensional surface adds dimensions.
- Cost. The search is exact, at n² × width.
PCA projections¶
FitPca() finds the directions along which a layer's representations of a set of inputs vary
most. PcaProject() places inputs on them, to plot, or to pass fewer dimensions on to another
method.
#include <fstream>
#include "pulsatrix/pca.hpp"
#include "pulsatrix/viz/svg.hpp"
// layer: (n, width), a row per input; words and tags: one per input.
const Pca pca = FitPca(layer, /*components=*/3);
const Tensor scores = PcaProject(pca, layer); // (n, 3)
const ProjectionDocument doc = MakeProjectionDocument(pca, scores, /*labels=*/words, /*groups=*/tags);
std::ofstream("layer.svg") << RenderProjectionSvg(doc); // PC1 against PC2
std::ofstream("layer13.svg") << RenderProjectionSvg(doc, 0, 2); // PC1 against PC3
std::ofstream("layer.json") << ToJson(doc); // for pulsatrix_svg or HTML
It matches scikit-learn's PCA to 1e-10:
- The data are centered.
- Variances divide by n − 1.
- Each component's largest entry is positive, so the same data always give the same picture.
It runs in double precision, through whichever is smaller of the covariance and the Gram matrix.
- Plots.
RenderProjectionSvgdraws two axes on one scale, so distances on the page are distances in the projection. Each axis is captioned with its share of the variance. - Points are colored by group or by value, and show their label on hover.
RenderProjectionHtmlis the zoomable version. With three components it shows the three pairs side by side, the views of a 3D projection.- The
pulsatrix.projection.v1document (ToJson) also lets any outside tool draw it. - Steering directions.
PcaComponent(pca, i)is component i as a unit vector forSteeringHook. One standard deviation along it issqrt(pca.explained_variance[i]), a natural unit for the coefficient. Then test whether it does anything withMeasureSteering(Steering). - Fewer dimensions.
PcaReconstructmaps coordinates back.PcaProject(pca, x, /*whiten=*/true)gives every axis variance 1.
Things to know:
- Check that the picture is stable. Components with close variances can swap or rotate
between two samples of the same inputs.
PcaSplitHalfOverlap(layer, k)fits each half of the inputs separately and compares the two k-dimensional spans: 1 means the same span, near k / width means unrelated. In the test, points on a plane give 0.9995, the same plane under noise as large as it gives 0.92, and noise alone gives 0.09. - Most variance isn't most meaning. In anisotropic layers the first component is often the common direction or a few outlier dimensions (see the shape of a layer). Look at the explained variance ratios, and treat a direction as a hypothesis to test.
SubspaceOverlap(a, b, k)compares any two PCAs of the same width, for example the same layer in two models, or before and after fine-tuning.
UMAP and t-SNE¶
pulsatrix doesn't run UMAP or t-SNE. It writes a layer's representations in the formats the tools that do run them read, and reads their coordinates back for its own plots.
#include "pulsatrix/embedding_export.hpp"
// layer: (n, width); words and tags: one per input. The directory must exist.
ExportEmbeddings("layer6", layer, {{"word", words}, {"tag", tags}});
// layer6/vectors.npy, layer6/vectors.tsv, layer6/metadata.tsv
- In Python:
np.load("layer6/vectors.npy")goes straight intoumap.UMAP().fit_transform,openTSNE.TSNE().fitor scikit-learn'sTSNE.WriteNpyandSerializeNpywrite any tensor the same way, float32 with its shape. - In the browser, with nothing to install: load
vectors.tsvandmetadata.tsvinto the Embedding Projector (projector.tensorflow.org), which runs UMAP, t-SNE and PCA and colors points by any metadata column. - Back in pulsatrix: save the coordinates with
np.save, read them withReadNpy, and make a projection document, which the PCA views draw:
const Tensor xy = ReadNpy("layer6/umap.npy", &backend); // (n, 2), from Python
const ProjectionDocument doc = MakeProjectionDocument(xy, "umap", words, tags);
std::ofstream("layer6_umap.svg") << RenderProjectionSvg(doc);
Things to know:
- These plots keep neighbors, not distances. The gaps between clusters, the clusters' sizes
and their densities carry little meaning, and shapes can come from the settings alone
(perplexity,
n_neighbors,min_dist). The views say so under every plot whose method isumap,tsne,t-sneoropentsne. Ask questions about distances of PCA, and about neighbors of nearest neighbors, which measures them directly. - Run them more than once. Both start from random positions. A cluster that appears in one run and not another, or under one perplexity and not another, isn't evidence of anything.
- Reduce first if the layer is wide. Running PCA to about 50 dimensions first (
FitPca,PcaProject, then export the scores) is standard for t-SNE, and makes both faster. - The reader takes what numpy writes for float arrays: float32 or float64, C or Fortran
order, format versions 1 to 3. It rejects anything else, including pickled objects, with an
error rather than a guess, and is fuzzed (
tools/fuzz/npy_fuzz.cpp).
Full API reference: Doxygen: Mechanistic Interpretability