Block-sparse featurizers (FEAT-6; Fel et al., "Structuring Sparsity: Block-Sparse
Featurizers Capture Visual Concept Manifolds", arXiv 2606.25234): features are small subspaces (blocks of directions) instead of single directions, so a concept that lives on a low-dimensional manifold, such as an angle or a lighting direction, is one feature.
More...
|
| struct | pulsatrix::BlockSparseOptions |
| |
| class | pulsatrix::BlockSparseFeaturizer |
| | A block-sparse featurizer: G blocks of b directions in a d-dimensional space. Codes are (N, G·b), block g's in columns [g·b, (g+1)·b), signed; the reconstruction is x̂ = z D, D's rows the atoms. A block is active when its codes aren't all zero, and its norm says how strongly; its coordinates say where on the concept's subspace the input is. More...
|
| |
| struct | pulsatrix::BlockGeometry |
| | Each block's code geometry over x, from the covariance of its codes on the inputs where it is active: stable rank ‖C‖²_F / ‖C‖²_2, participation ratio (tr C)² / ‖C‖²_F and effective rank (the exponential of the eigenvalues' entropy). The paper finds a block's dimension saturates at the concept's (2 to 4 for DINOv3) however large b is. More...
|
| |
| struct | pulsatrix::DescriptionLength |
| | The paper's minimum description length (Appendix C), in bits per input at distortion δ = distortion_fraction · (mean per-dimension variance of x): More...
|
| |
Block-sparse featurizers (FEAT-6; Fel et al., "Structuring Sparsity: Block-Sparse
Featurizers Capture Visual Concept Manifolds", arXiv 2606.25234): features are small subspaces (blocks of directions) instead of single directions, so a concept that lives on a low-dimensional manifold, such as an angle or a lighting direction, is one feature.
- Note
- Implemented after the paper and its reference code (github.com/goodfire-ai/ block-sparse-featurizer). Where the two differ, the code is followed: a γ per block in the Grassmannian variant, and a block JumpReLU (not a soft threshold) in the group-lasso variant. Tournament top-k is after Jerpelea and Ananthram, arXiv 2608.27515.