pulsatrix 1.1.0
Loading...
Searching...
No Matches
featurizer.hpp File Reference

The featurizer family's common interface (FEAT-1): models that rewrite activations as sparse codes over learned directions and back, such as sparse autoencoders, with a shared training step and metrics (L0, dead and dense latents). More...

#include <cstdint>
#include <vector>
#include "pulsatrix/module.hpp"
#include "pulsatrix/tensor.hpp"
Include dependency graph for featurizer.hpp:
This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes

struct  pulsatrix::FeaturizerLoss
 One batch's loss, split into its parts. More...
 
class  pulsatrix::Featurizer
 A featurizer maps activations x, (N, input_dim), to codes f = encode(x), (N, num_features), and reconstructs x ≈ decode(f) = f D + b: feature i writes along its decoder direction D[i]. More...
 
class  pulsatrix::FeatureActivityTracker
 How often each feature fires, and how long since it last did. A feature that hasn't fired in a long window is dead: it no longer learns, and wastes capacity. One that fires on most inputs is dense, usually a sign it encodes a bias, not a feature. More...
 

Namespaces

namespace  pulsatrix
 

Functions

double pulsatrix::MeanL0 (const std::vector< float > &codes, int64_t num_features, float threshold=0.0f)
 The mean number of codes above threshold in magnitude per input (L0), for codes row-major (N, num_features).
 
double pulsatrix::MeanL0 (Featurizer &featurizer, const Tensor &x, float threshold=0.0f)
 MeanL0 of a featurizer's codes for x, in blocks: a block counts once when any of its codes is above threshold in magnitude.
 
template<typename Optimizer >
FeaturizerLoss pulsatrix::TrainFeaturizer (Featurizer &featurizer, const Tensor &x, Optimizer &optimizer, bool unit_norm_decoder=true, FeatureActivityTracker *tracker=nullptr)
 One training step for any featurizer: zero the gradients, loss_and_backward(), one optimizer step, then normalize_decoder() when unit_norm_decoder. When tracker is given it observes the batch's codes.
 
template<typename Optimizer >
FeaturizerLoss pulsatrix::TrainFeaturizer (Featurizer &featurizer, const Tensor &x, const Tensor &target, Optimizer &optimizer, bool unit_norm_decoder=true, FeatureActivityTracker *tracker=nullptr)
 TrainFeaturizer for a transcoder: one step on predicting target from x.
 

Detailed Description

The featurizer family's common interface (FEAT-1): models that rewrite activations as sparse codes over learned directions and back, such as sparse autoencoders, with a shared training step and metrics (L0, dead and dense latents).