pulsatrix 1.1.0
Loading...
Searching...
No Matches
block_sparse_featurizer.hpp File Reference

Block-sparse featurizers (FEAT-6; Fel et al., "Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds", arXiv 2606.25234): features are small subspaces (blocks of directions) instead of single directions, so a concept that lives on a low-dimensional manifold, such as an angle or a lighting direction, is one feature. More...

#include <cstdint>
#include <map>
#include <optional>
#include <utility>
#include <vector>
#include "pulsatrix/featurizer.hpp"
#include "pulsatrix/linear_module.hpp"
Include dependency graph for block_sparse_featurizer.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::BlockSparseOptions
 
class  pulsatrix::BlockSparseFeaturizer
 A block-sparse featurizer: G blocks of b directions in a d-dimensional space. Codes are (N, G·b), block g's in columns [g·b, (g+1)·b), signed; the reconstruction is x̂ = z D, D's rows the atoms. A block is active when its codes aren't all zero, and its norm says how strongly; its coordinates say where on the concept's subspace the input is. More...
 
struct  pulsatrix::BlockGeometry
 Each block's code geometry over x, from the covariance of its codes on the inputs where it is active: stable rank ‖C‖²_F / ‖C‖²_2, participation ratio (tr C)² / ‖C‖²_F and effective rank (the exponential of the eigenvalues' entropy). The paper finds a block's dimension saturates at the concept's (2 to 4 for DINOv3) however large b is. More...
 
struct  pulsatrix::DescriptionLength
 The paper's minimum description length (Appendix C), in bits per input at distortion δ = distortion_fraction · (mean per-dimension variance of x): More...
 

Namespaces

namespace  pulsatrix
 

Enumerations

enum class  pulsatrix::BsfVariant { pulsatrix::Vanilla , pulsatrix::Grassmannian , pulsatrix::GroupLasso }
 
enum class  pulsatrix::BlockSelection { pulsatrix::TopK , pulsatrix::Tournament }
 

Functions

BlockGeometry pulsatrix::MeasureBlockGeometry (Featurizer &featurizer, const Tensor &x, int64_t batch=4096)
 
DescriptionLength pulsatrix::MeasureDescriptionLength (Featurizer &featurizer, const Tensor &x, double distortion_fraction=0.1, int64_t batch=4096)
 

Detailed Description

Block-sparse featurizers (FEAT-6; Fel et al., "Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds", arXiv 2606.25234): features are small subspaces (blocks of directions) instead of single directions, so a concept that lives on a low-dimensional manifold, such as an angle or a lighting direction, is one feature.

Note
Implemented after the paper and its reference code (github.com/goodfire-ai/ block-sparse-featurizer). Where the two differ, the code is followed: a γ per block in the Grassmannian variant, and a block JumpReLU (not a soft threshold) in the group-lasso variant. Tournament top-k is after Jerpelea and Ananthram, arXiv 2608.27515.