|
pulsatrix 1.1.0
|
The featurizer family's common interface (FEAT-1): models that rewrite activations as sparse codes over learned directions and back, such as sparse autoencoders, with a shared training step and metrics (L0, dead and dense latents). More...
#include <cstdint>#include <vector>#include "pulsatrix/module.hpp"#include "pulsatrix/tensor.hpp"

Go to the source code of this file.
Classes | |
| struct | pulsatrix::FeaturizerLoss |
| One batch's loss, split into its parts. More... | |
| class | pulsatrix::Featurizer |
A featurizer maps activations x, (N, input_dim), to codes f = encode(x), (N, num_features), and reconstructs x ≈ decode(f) = f D + b: feature i writes along its decoder direction D[i]. More... | |
| class | pulsatrix::FeatureActivityTracker |
| How often each feature fires, and how long since it last did. A feature that hasn't fired in a long window is dead: it no longer learns, and wastes capacity. One that fires on most inputs is dense, usually a sign it encodes a bias, not a feature. More... | |
Namespaces | |
| namespace | pulsatrix |
Functions | |
| double | pulsatrix::MeanL0 (const std::vector< float > &codes, int64_t num_features, float threshold=0.0f) |
The mean number of codes above threshold in magnitude per input (L0), for codes row-major (N, num_features). | |
| double | pulsatrix::MeanL0 (Featurizer &featurizer, const Tensor &x, float threshold=0.0f) |
MeanL0 of a featurizer's codes for x, in blocks: a block counts once when any of its codes is above threshold in magnitude. | |
| template<typename Optimizer > | |
| FeaturizerLoss | pulsatrix::TrainFeaturizer (Featurizer &featurizer, const Tensor &x, Optimizer &optimizer, bool unit_norm_decoder=true, FeatureActivityTracker *tracker=nullptr) |
One training step for any featurizer: zero the gradients, loss_and_backward(), one optimizer step, then normalize_decoder() when unit_norm_decoder. When tracker is given it observes the batch's codes. | |
| template<typename Optimizer > | |
| FeaturizerLoss | pulsatrix::TrainFeaturizer (Featurizer &featurizer, const Tensor &x, const Tensor &target, Optimizer &optimizer, bool unit_norm_decoder=true, FeatureActivityTracker *tracker=nullptr) |
TrainFeaturizer for a transcoder: one step on predicting target from x. | |
The featurizer family's common interface (FEAT-1): models that rewrite activations as sparse codes over learned directions and back, such as sparse autoencoders, with a shared training step and metrics (L0, dead and dense latents).