107 [[nodiscard]] std::vector<float>
gamma()
const;
115 [[nodiscard]] int64_t
input_dim()
const override {
return d_; }
117 [[nodiscard]] int64_t
num_features()
const override {
return G_ * b_; }
118 [[nodiscard]] int64_t
block_size()
const override {
return b_; }
144 std::vector<float> a;
145 std::vector<float> z;
146 std::vector<float> gate;
147 std::vector<float> norm;
149 Pass Run(
const Tensor& x,
bool training);
150 void Check(
const Tensor& x,
const char* who)
const;
152 void SyncTiedEncoder();
154 std::vector<int64_t> Select(
const std::vector<float>& norms,
const std::vector<std::vector<float>>* frames,
bool training);
156 std::vector<std::vector<float>> Frames();
159 Tensor BackwardThroughEncoder(
const Pass& p,
const std::vector<float>& g_z,
double l0_weight,
bool straight_through);
166 Tensor gamma_, gamma_grad_;
167 Tensor threshold_raw_, threshold_raw_grad_;
168 Tensor bandwidth_, log_lambda_, initialized_;
170 std::vector<int64_t> wins_;
171 std::map<int64_t, int64_t> lost_to_;
173 int64_t last_rows_ = 0;
215 int64_t batch = 4096);
A block-sparse featurizer: G blocks of b directions in a d-dimensional space. Codes are (N,...
Definition block_sparse_featurizer.hpp:96
int64_t block_size() const override
Codes come in blocks of this many (FEAT-6): a block-sparse featurizer's feature is a block,...
Definition block_sparse_featurizer.hpp:118
void set_training(bool training) override
Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Mod...
std::vector< NamedBufferRef > named_buffers() override
This module's buffers: state a checkpoint must save that no optimizer updates.
int64_t input_dim() const override
Definition block_sparse_featurizer.hpp:115
BlockSparseFeaturizer(int64_t dim, int64_t num_blocks, DeviceBackend *backend, BlockSparseOptions options={})
LinearModule & encoder()
Definition block_sparse_featurizer.hpp:111
OpType op_type() const override
This module's operation-category tag, for ComputationGraph node tagging.
Definition block_sparse_featurizer.hpp:131
std::vector< float > thresholds() const
GroupLasso's thresholds θ per block, and the current λ.
void release_activations() override
Frees the tensors forward() kept for backward() and the relevance passes; containers release their ch...
void normalize_decoder() override
Rescales every feature so its decoder direction has unit L2 norm, scaling its code by the inverse whe...
Tensor propagate_relevance(const Tensor &relevance_out, const LRPRuleConfig &config) override
Computes this module's contribution to LRP relevance propagation.
std::vector< float > decoder_direction(int64_t i) override
Atom i (block i / b, direction i % b), d values.
Module & parameters_module() override
The trainable parameters, for an optimizer's step() and zero_grad(), and for checkpoints.
Definition block_sparse_featurizer.hpp:125
float l0_multiplier() const
int64_t num_features() const override
The codes per input, num_blocks · block_size.
Definition block_sparse_featurizer.hpp:117
Tensor decode(const Tensor &codes) override
The reconstruction from codes, (N, output_dim).
Tensor backward(const Tensor &grad_output) override
Backward with the selection held fixed.
Tensor encode(const Tensor &x) override
The codes for a batch, (N, num_features).
std::vector< float > gamma() const
The Grassmannian γ per block (empty for the other variants).
LinearModule & decoder()
Definition block_sparse_featurizer.hpp:112
std::optional< DeviceType > compute_device() const override
The device this module computes on, so forward() can reject an input on another device before any ker...
Definition block_sparse_featurizer.hpp:134
Tensor forward_impl(const Tensor &input) override
std::vector< float > block_frame(int64_t g)
Block g's frame, b x d row-major (its b atoms).
std::vector< NamedParamRef > named_parameters() override
This module's trainable parameters, each with its hierarchical name – the one place a module declares...
int64_t num_blocks() const
Definition block_sparse_featurizer.hpp:102
FeaturizerLoss loss_and_backward(const Tensor &x, std::vector< float > *codes=nullptr) override
Computes the training loss on x and adds its gradients to the parameters' (zero them first)....
const BlockSparseOptions & options() const
Definition block_sparse_featurizer.hpp:103
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
virtual DeviceType device() const noexcept=0
Which device this backend's buffers reside on.
A featurizer maps activations x, (N, input_dim), to codes f = encode(x), (N, num_features),...
Definition featurizer.hpp:45
y = x @ W + b, batched (x is (N, in_features), y is (N, out_features)) – migrated from the original u...
Definition linear_module.hpp:43
Base class for every layer type (LinearModule, Conv2DModule, activations, ...).
Definition module.hpp:78
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
The featurizer family's common interface (FEAT-1): models that rewrite activations as sparse codes ov...
Dense/fully-connected layer – the reference Module implementation.
Definition acquisition_functions.hpp:16
BsfVariant
Definition block_sparse_featurizer.hpp:25
@ Grassmannian
Tied, orthonormal frames: z_g = γ_g x D_gᵀ with D_g D_gᵀ = I, then Π_k.
@ Vanilla
A free encoder: z = Π_k(x W_enc + b_enc), decoder atoms of unit norm.
@ GroupLasso
A free encoder and a learned threshold per block: a block fires when its norm is above θ_g,...
BlockGeometry MeasureBlockGeometry(Featurizer &featurizer, const Tensor &x, int64_t batch=4096)
BlockSelection
Definition block_sparse_featurizer.hpp:35
@ TopK
Π_k: each input keeps its k blocks of largest norm.
@ Tournament
Tournament top-k (Jerpelea and Ananthram): blocks are taken by norm, but one that overlaps an already...
DescriptionLength MeasureDescriptionLength(Featurizer &featurizer, const Tensor &x, double distortion_fraction=0.1, int64_t batch=4096)
OpType
The op-type tag a Node carries. Charter Part 2 §3: nodes are tagged by a small closed set of op types...
Definition op_type.hpp:19
Each block's code geometry over x, from the covariance of its codes on the inputs where it is active:...
Definition block_sparse_featurizer.hpp:182
double mean_effective_rank
Definition block_sparse_featurizer.hpp:188
std::vector< double > firing_rate
per block
Definition block_sparse_featurizer.hpp:183
double mean_participation_ratio
Definition block_sparse_featurizer.hpp:188
std::vector< double > effective_rank
Definition block_sparse_featurizer.hpp:186
std::vector< double > participation_ratio
Definition block_sparse_featurizer.hpp:185
std::vector< double > stable_rank
per block; 0 where it fired on fewer than 2 inputs
Definition block_sparse_featurizer.hpp:184
double mean_stable_rank
Means over blocks that fired on at least 2 inputs, weighted by firing rate.
Definition block_sparse_featurizer.hpp:188
Definition block_sparse_featurizer.hpp:43
int64_t block_size
b, the directions per block (the paper's main runs use 3).
Definition block_sparse_featurizer.hpp:46
int64_t k
Blocks kept per input (Vanilla, Grassmannian).
Definition block_sparse_featurizer.hpp:48
int64_t tournament_warmup
Training steps of plain top-k before duels begin.
Definition block_sparse_featurizer.hpp:57
double verdict_overlap
Definition block_sparse_featurizer.hpp:55
float dual_lr
Definition block_sparse_featurizer.hpp:62
BsfVariant variant
Definition block_sparse_featurizer.hpp:44
double duel_overlap
Tournament: a candidate block duels an accepted one when their subspace overlap |Q_g Q_hᵀ|²_F / b exc...
Definition block_sparse_featurizer.hpp:54
BlockSelection selection
Definition block_sparse_featurizer.hpp:49
float gain
GroupLasso: θ_g = softplus(gain · raw_g); the straight-through Gaussian kernel's bandwidth is this mu...
Definition block_sparse_featurizer.hpp:65
uint64_t seed
Definition block_sparse_featurizer.hpp:67
double target_l0
GroupLasso: the mean number of active blocks per input to hold, by dual ascent on log λ with step dua...
Definition block_sparse_featurizer.hpp:60
float bandwidth
Definition block_sparse_featurizer.hpp:66
float initial_lambda
Definition block_sparse_featurizer.hpp:61
The paper's minimum description length (Appendix C), in bits per input at distortion δ = distortion_f...
Definition block_sparse_featurizer.hpp:207
double delta
Definition block_sparse_featurizer.hpp:209
double dictionary
Definition block_sparse_featurizer.hpp:208
double total
Definition block_sparse_featurizer.hpp:208
double support
Definition block_sparse_featurizer.hpp:208
double code
Definition block_sparse_featurizer.hpp:208
double residual
Definition block_sparse_featurizer.hpp:208
One batch's loss, split into its parts.
Definition featurizer.hpp:18
Configuration for LRP relevance propagation: which rule a module applies and its hyperparameters....
Definition lrp_rule_config.hpp:72