|
pulsatrix 1.1.0
|
A featurizer maps activations x, (N, input_dim), to codes f = encode(x), (N, num_features), and reconstructs x ≈ decode(f) = f D + b: feature i writes along its decoder direction D[i].
More...
#include <featurizer.hpp>

Public Member Functions | |
| virtual | ~Featurizer ()=default |
| virtual int64_t | input_dim () const =0 |
| virtual int64_t | num_features () const =0 |
| virtual int64_t | output_dim () const |
| What decode() produces: input_dim() for an autoencoder. | |
| virtual int64_t | block_size () const |
| Codes come in blocks of this many (FEAT-6): a block-sparse featurizer's feature is a block, active when any of its codes is nonzero, and its codes may be negative. 1 (the default) for featurizers whose features are single directions. | |
| virtual Tensor | encode (const Tensor &x)=0 |
The codes for a batch, (N, num_features). | |
| virtual Tensor | decode (const Tensor &codes)=0 |
The reconstruction from codes, (N, output_dim). | |
| virtual Tensor | predict (const Tensor &x) |
The full output for a batch, (N, output_dim): decode(encode(x)), plus any part that bypasses the codes (a transcoder's skip connection). | |
| virtual FeaturizerLoss | loss_and_backward (const Tensor &x, std::vector< float > *codes=nullptr)=0 |
Computes the training loss on x and adds its gradients to the parameters' (zero them first). When codes is given, it receives the batch's codes, row-major (N, num_features), for metrics. | |
| virtual FeaturizerLoss | loss_and_backward_with_target (const Tensor &x, const Tensor &target, std::vector< float > *codes=nullptr) |
The loss for predicting target, (N, output_dim), from x: a transcoder's. | |
| virtual void | normalize_decoder ()=0 |
| Rescales every feature so its decoder direction has unit L2 norm, scaling its code by the inverse where the encoder allows it, so reconstructions don't change. Training with unit-norm decoders keeps a sparsity penalty from being dodged by shrinking codes and growing directions, and makes code sizes comparable across features. | |
| virtual std::vector< float > | decoder_direction (int64_t i)=0 |
Feature i's decoder direction, output_dim values. | |
| virtual Module & | parameters_module ()=0 |
The trainable parameters, for an optimizer's step() and zero_grad(), and for checkpoints. | |
A featurizer maps activations x, (N, input_dim), to codes f = encode(x), (N, num_features), and reconstructs x ≈ decode(f) = f D + b: feature i writes along its decoder direction D[i].
Sparse autoencoders (SparseAutoencoder) were the first. TopK, BatchTopK and Matryoshka (TopKSparseAutoencoder) and JumpReLU (JumpReLUSparseAutoencoder) follow, and transcoders and block-sparse featurizers (FEAT-5, FEAT-6) will too, so training, metrics and views work for all of them.
A transcoder (FEAT-5) predicts another activation instead of reconstructing its input: an MLP's output from its input. It has an output_dim(), predict() adds what doesn't go through the codes (a skip connection), and it trains with loss_and_backward_with_target().
|
virtualdefault |
|
inlinevirtual |
Codes come in blocks of this many (FEAT-6): a block-sparse featurizer's feature is a block, active when any of its codes is nonzero, and its codes may be negative. 1 (the default) for featurizers whose features are single directions.
Reimplemented in pulsatrix::BlockSparseFeaturizer.
The reconstruction from codes, (N, output_dim).
Implemented in pulsatrix::BlockSparseFeaturizer, pulsatrix::Crosscoder, pulsatrix::JumpReLUSparseAutoencoder, pulsatrix::SparseAutoencoder, pulsatrix::TopKSparseAutoencoder, and pulsatrix::Transcoder.
|
pure virtual |
Feature i's decoder direction, output_dim values.
Implemented in pulsatrix::BlockSparseFeaturizer, pulsatrix::Crosscoder, pulsatrix::JumpReLUSparseAutoencoder, pulsatrix::SparseAutoencoder, pulsatrix::TopKSparseAutoencoder, and pulsatrix::Transcoder.
The codes for a batch, (N, num_features).
Implemented in pulsatrix::BlockSparseFeaturizer, pulsatrix::Crosscoder, pulsatrix::JumpReLUSparseAutoencoder, pulsatrix::SparseAutoencoder, pulsatrix::TopKSparseAutoencoder, and pulsatrix::Transcoder.
|
pure virtual |
|
pure virtual |
Computes the training loss on x and adds its gradients to the parameters' (zero them first). When codes is given, it receives the batch's codes, row-major (N, num_features), for metrics.
| std::invalid_argument | for a batch that isn't (N, input_dim) with N > 0. |
Implemented in pulsatrix::SparseAutoencoder, pulsatrix::BlockSparseFeaturizer, pulsatrix::Crosscoder, pulsatrix::JumpReLUSparseAutoencoder, pulsatrix::TopKSparseAutoencoder, and pulsatrix::Transcoder.
|
virtual |
The loss for predicting target, (N, output_dim), from x: a transcoder's.
| std::invalid_argument | for an autoencoder, whose target is its input (the default). |
Reimplemented in pulsatrix::Transcoder.
|
pure virtual |
Rescales every feature so its decoder direction has unit L2 norm, scaling its code by the inverse where the encoder allows it, so reconstructions don't change. Training with unit-norm decoders keeps a sparsity penalty from being dodged by shrinking codes and growing directions, and makes code sizes comparable across features.
Implemented in pulsatrix::BlockSparseFeaturizer, pulsatrix::Crosscoder, pulsatrix::JumpReLUSparseAutoencoder, pulsatrix::SparseAutoencoder, pulsatrix::TopKSparseAutoencoder, and pulsatrix::Transcoder.
|
pure virtual |
|
inlinevirtual |
What decode() produces: input_dim() for an autoencoder.
Reimplemented in pulsatrix::Transcoder.
|
pure virtual |
The trainable parameters, for an optimizer's step() and zero_grad(), and for checkpoints.
Implemented in pulsatrix::BlockSparseFeaturizer, pulsatrix::Crosscoder, pulsatrix::JumpReLUSparseAutoencoder, pulsatrix::SparseAutoencoder, pulsatrix::TopKSparseAutoencoder, and pulsatrix::Transcoder.
The full output for a batch, (N, output_dim): decode(encode(x)), plus any part that bypasses the codes (a transcoder's skip connection).
Reimplemented in pulsatrix::Transcoder.