|
pulsatrix 1.1.0
|
One number per layer for the shape of its representations: anisotropy, effective rank and TwoNN intrinsic dimension. More...

Go to the source code of this file.
Classes | |
| struct | pulsatrix::EffectiveRank |
| MeasureEffectiveRank()'s result. More... | |
| struct | pulsatrix::TwoNNResult |
| TwoNNDimension()'s result. More... | |
Namespaces | |
| namespace | pulsatrix |
Functions | |
| double | pulsatrix::Anisotropy (const Tensor &x, bool center=false) |
| The mean cosine similarity over all pairs of different rows: about 0 for directions spread evenly, near 1 when every row points the same way. | |
| EffectiveRank | pulsatrix::MeasureEffectiveRank (const Tensor &x, bool center=true) |
| How many directions the rows spread along, from the singular values of x. | |
| TwoNNResult | pulsatrix::TwoNNDimension (const Tensor &x, double discard_fraction=0.1) |
| TwoNN intrinsic dimension (Facco, d'Errico, Rodriguez and Laio, Scientific Reports 2017): from each point's distances to its two nearest neighbors, mu = r2 / r1, whose distribution depends only on the dimension the points vary in locally. | |
One number per layer for the shape of its representations: anisotropy, effective rank and TwoNN intrinsic dimension.
Language models' representations are anisotropic (Ethayarajh, EMNLP 2019): they share a few large common directions, so the cosine similarity of two unrelated inputs is high, often 0.5 or more, and raw cosine similarity says little about meaning. Anisotropy() measures that, and shows how much centering removes. Effective rank says how many directions a layer spreads its inputs along, and TwoNN how many dimensions the inputs actually vary in locally, which can be far fewer than the width.
Everything is computed in double precision on the host.