pulsatrix 1.1.0
Loading...
Searching...
No Matches
representation_geometry.hpp File Reference

One number per layer for the shape of its representations: anisotropy, effective rank and TwoNN intrinsic dimension. More...

#include <cstdint>
#include <vector>
#include "pulsatrix/tensor.hpp"
Include dependency graph for representation_geometry.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::EffectiveRank
 MeasureEffectiveRank()'s result. More...
 
struct  pulsatrix::TwoNNResult
 TwoNNDimension()'s result. More...
 

Namespaces

namespace  pulsatrix
 

Functions

double pulsatrix::Anisotropy (const Tensor &x, bool center=false)
 The mean cosine similarity over all pairs of different rows: about 0 for directions spread evenly, near 1 when every row points the same way.
 
EffectiveRank pulsatrix::MeasureEffectiveRank (const Tensor &x, bool center=true)
 How many directions the rows spread along, from the singular values of x.
 
TwoNNResult pulsatrix::TwoNNDimension (const Tensor &x, double discard_fraction=0.1)
 TwoNN intrinsic dimension (Facco, d'Errico, Rodriguez and Laio, Scientific Reports 2017): from each point's distances to its two nearest neighbors, mu = r2 / r1, whose distribution depends only on the dimension the points vary in locally.
 

Detailed Description

One number per layer for the shape of its representations: anisotropy, effective rank and TwoNN intrinsic dimension.

Language models' representations are anisotropic (Ethayarajh, EMNLP 2019): they share a few large common directions, so the cosine similarity of two unrelated inputs is high, often 0.5 or more, and raw cosine similarity says little about meaning. Anisotropy() measures that, and shows how much centering removes. Effective rank says how many directions a layer spreads its inputs along, and TwoNN how many dimensions the inputs actually vary in locally, which can be far fewer than the width.

Everything is computed in double precision on the host.