pulsatrix 1.1.0
Loading...
Searching...
No Matches
representation_similarity.hpp File Reference

How similar two layers' representations of the same inputs are: linear CKA, RSA and SVCCA. More...

#include <cstddef>
#include <cstdint>
#include <vector>
#include "pulsatrix/tensor.hpp"
Include dependency graph for representation_similarity.hpp:
This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes

struct  pulsatrix::CkaMatrix
 CKA between every pair from two lists of representations, such as the layers of two models: values[i * cols + j] compares a[i] with b[j]. More...
 
struct  pulsatrix::Rdm
 A representational dissimilarity matrix: the distance between every pair of n inputs. More...
 
struct  pulsatrix::RsaTestResult
 RsaPermutationTest()'s result. More...
 
struct  pulsatrix::SvccaOptions
 Svcca()'s settings. More...
 
struct  pulsatrix::SvccaResult
 Svcca()'s result. More...
 

Namespaces

namespace  pulsatrix
 

Enumerations

enum class  pulsatrix::CkaEstimator { pulsatrix::Debiased , pulsatrix::Biased }
 Which estimator of CKA to use. More...
 
enum class  pulsatrix::RdmMetric { pulsatrix::Correlation , pulsatrix::Euclidean , pulsatrix::Cosine }
 How RepresentationalDissimilarity() measures the distance between two inputs. More...
 
enum class  pulsatrix::RdmComparison { pulsatrix::Spearman , pulsatrix::Pearson }
 How CompareRdms() correlates two RDMs. More...
 

Functions

double pulsatrix::LinearCka (const Tensor &x, const Tensor &y, CkaEstimator estimator=CkaEstimator::Debiased)
 Linear CKA between two representations of the same inputs.
 
CkaMatrix pulsatrix::LinearCkaMatrix (const std::vector< Tensor > &a, const std::vector< Tensor > &b, CkaEstimator estimator=CkaEstimator::Debiased, std::size_t max_gram_bytes=std::size_t{2}<< 30)
 LinearCka() between every representation in a and every one in b.
 
Rdm pulsatrix::RepresentationalDissimilarity (const Tensor &x, RdmMetric metric=RdmMetric::Correlation)
 The RDM of a representation, shape (n, ...): a row per input, later dimensions flattened.
 
double pulsatrix::CompareRdms (const Rdm &a, const Rdm &b, RdmComparison comparison=RdmComparison::Spearman)
 The correlation between two RDMs of the same inputs, in the same order.
 
double pulsatrix::Rsa (const Tensor &x, const Tensor &y, RdmMetric metric=RdmMetric::Correlation, RdmComparison comparison=RdmComparison::Spearman)
 RSA between two representations of the same inputs: CompareRdms(RepresentationalDissimilarity(x), RepresentationalDissimilarity(y)).
 
RsaTestResult pulsatrix::RsaPermutationTest (const Rdm &a, const Rdm &b, RdmComparison comparison=RdmComparison::Spearman, int64_t permutations=999, uint64_t seed=0)
 Whether two RDMs agree more than chance: the Mantel test RSA uses.
 
std::vector< double > pulsatrix::CanonicalCorrelations (const Tensor &x, const Tensor &y)
 The canonical correlations between two representations of the same inputs, largest first: min(rank x, rank y) values between 0 and 1.
 
SvccaResult pulsatrix::Svcca (const Tensor &x, const Tensor &y, const SvccaOptions &options={})
 SVCCA: CanonicalCorrelations() between each representation's top singular directions.
 

Detailed Description

How similar two layers' representations of the same inputs are: linear CKA, RSA and SVCCA.

Centered kernel alignment (Kornblith et al., ICML 2019) compares two representations of the same n inputs, for example one layer of two models, two layers of one model, a model before and after fine-tuning, or the same layer computed on two backends. It is 1 when one representation is a rotation and uniform scaling of the other, and near 0 when they are unrelated. Unlike comparing activations directly, it doesn't need the layers to have the same width or their neurons to line up.

The debiased estimator (Song et al., JMLR 2012; Nguyen et al., ICLR 2021) is the default. The original, biased one is pulled towards 1 when there are few inputs for the width: two random, unrelated representations of 50 inputs with 1,000 features each score 0.95 with it and about 0 debiased.

Two older measures are here too, for comparison with published results that use them:

  • RSA (representational similarity analysis; Kriegeskorte et al., Frontiers in Systems Neuroscience 2008) describes each representation by how far apart it puts every pair of inputs, its representational dissimilarity matrix (RDM), and correlates the two RDMs. It compares a model with anything that orders the same inputs, a brain recording or a hand-made model of the categories, and RsaPermutationTest() says whether the agreement could be chance.
  • SVCCA (Raghu et al., NeurIPS 2017) keeps each representation's top singular directions (99% of the variance by default) and finds the pairs of directions, one in each, that correlate most. It is invariant to any invertible linear map, which CKA isn't. That makes it blind to how a layer weights its directions, and Kornblith et al. found it worse than CKA at matching corresponding layers. It is also biased upwards: canonical correlations are fitted to the inputs, so unrelated representations score well above 0 unless there are many more inputs than kept directions.