pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::Featurizer Class Referenceabstract

A featurizer maps activations x, (N, input_dim), to codes f = encode(x), (N, num_features), and reconstructs x ≈ decode(f) = f D + b: feature i writes along its decoder direction D[i]. More...

#include <featurizer.hpp>

Inheritance diagram for pulsatrix::Featurizer:

Public Member Functions

virtual ~Featurizer ()=default
 
virtual int64_t input_dim () const =0
 
virtual int64_t num_features () const =0
 
virtual int64_t output_dim () const
 What decode() produces: input_dim() for an autoencoder.
 
virtual int64_t block_size () const
 Codes come in blocks of this many (FEAT-6): a block-sparse featurizer's feature is a block, active when any of its codes is nonzero, and its codes may be negative. 1 (the default) for featurizers whose features are single directions.
 
virtual Tensor encode (const Tensor &x)=0
 The codes for a batch, (N, num_features).
 
virtual Tensor decode (const Tensor &codes)=0
 The reconstruction from codes, (N, output_dim).
 
virtual Tensor predict (const Tensor &x)
 The full output for a batch, (N, output_dim): decode(encode(x)), plus any part that bypasses the codes (a transcoder's skip connection).
 
virtual FeaturizerLoss loss_and_backward (const Tensor &x, std::vector< float > *codes=nullptr)=0
 Computes the training loss on x and adds its gradients to the parameters' (zero them first). When codes is given, it receives the batch's codes, row-major (N, num_features), for metrics.
 
virtual FeaturizerLoss loss_and_backward_with_target (const Tensor &x, const Tensor &target, std::vector< float > *codes=nullptr)
 The loss for predicting target, (N, output_dim), from x: a transcoder's.
 
virtual void normalize_decoder ()=0
 Rescales every feature so its decoder direction has unit L2 norm, scaling its code by the inverse where the encoder allows it, so reconstructions don't change. Training with unit-norm decoders keeps a sparsity penalty from being dodged by shrinking codes and growing directions, and makes code sizes comparable across features.
 
virtual std::vector< float > decoder_direction (int64_t i)=0
 Feature i's decoder direction, output_dim values.
 
virtual Module & parameters_module ()=0
 The trainable parameters, for an optimizer's step() and zero_grad(), and for checkpoints.
 

Detailed Description

A featurizer maps activations x, (N, input_dim), to codes f = encode(x), (N, num_features), and reconstructs x ≈ decode(f) = f D + b: feature i writes along its decoder direction D[i].

Sparse autoencoders (SparseAutoencoder) were the first. TopK, BatchTopK and Matryoshka (TopKSparseAutoencoder) and JumpReLU (JumpReLUSparseAutoencoder) follow, and transcoders and block-sparse featurizers (FEAT-5, FEAT-6) will too, so training, metrics and views work for all of them.

A transcoder (FEAT-5) predicts another activation instead of reconstructing its input: an MLP's output from its input. It has an output_dim(), predict() adds what doesn't go through the codes (a skip connection), and it trains with loss_and_backward_with_target().

Note
A featurizer is a discovery tool: a low reconstruction error says nothing about whether its directions are meaningful or causal. Compare features with probes and baselines.

Constructor & Destructor Documentation

◆ ~Featurizer()

virtual pulsatrix::Featurizer::~Featurizer ( )
virtualdefault

Member Function Documentation

◆ block_size()

virtual int64_t pulsatrix::Featurizer::block_size ( ) const
inlinevirtual

Codes come in blocks of this many (FEAT-6): a block-sparse featurizer's feature is a block, active when any of its codes is nonzero, and its codes may be negative. 1 (the default) for featurizers whose features are single directions.

Reimplemented in pulsatrix::BlockSparseFeaturizer.

◆ decode()

virtual Tensor pulsatrix::Featurizer::decode ( const Tensor &  codes)
pure virtual

◆ decoder_direction()

virtual std::vector< float > pulsatrix::Featurizer::decoder_direction ( int64_t  i)
pure virtual

◆ encode()

virtual Tensor pulsatrix::Featurizer::encode ( const Tensor &  x)
pure virtual

◆ input_dim()

◆ loss_and_backward()

virtual FeaturizerLoss pulsatrix::Featurizer::loss_and_backward ( const Tensor &  x,
std::vector< float > *  codes = nullptr 
)
pure virtual

Computes the training loss on x and adds its gradients to the parameters' (zero them first). When codes is given, it receives the batch's codes, row-major (N, num_features), for metrics.

Exceptions
std::invalid_argumentfor a batch that isn't (N, input_dim) with N > 0.

Implemented in pulsatrix::SparseAutoencoder, pulsatrix::BlockSparseFeaturizer, pulsatrix::Crosscoder, pulsatrix::JumpReLUSparseAutoencoder, pulsatrix::TopKSparseAutoencoder, and pulsatrix::Transcoder.

◆ loss_and_backward_with_target()

virtual FeaturizerLoss pulsatrix::Featurizer::loss_and_backward_with_target ( const Tensor &  x,
const Tensor &  target,
std::vector< float > *  codes = nullptr 
)
virtual

The loss for predicting target, (N, output_dim), from x: a transcoder's.

Exceptions
std::invalid_argumentfor an autoencoder, whose target is its input (the default).

Reimplemented in pulsatrix::Transcoder.

◆ normalize_decoder()

virtual void pulsatrix::Featurizer::normalize_decoder ( )
pure virtual

Rescales every feature so its decoder direction has unit L2 norm, scaling its code by the inverse where the encoder allows it, so reconstructions don't change. Training with unit-norm decoders keeps a sparsity penalty from being dodged by shrinking codes and growing directions, and makes code sizes comparable across features.

Implemented in pulsatrix::BlockSparseFeaturizer, pulsatrix::Crosscoder, pulsatrix::JumpReLUSparseAutoencoder, pulsatrix::SparseAutoencoder, pulsatrix::TopKSparseAutoencoder, and pulsatrix::Transcoder.

◆ num_features()

◆ output_dim()

virtual int64_t pulsatrix::Featurizer::output_dim ( ) const
inlinevirtual

What decode() produces: input_dim() for an autoencoder.

Reimplemented in pulsatrix::Transcoder.

◆ parameters_module()

virtual Module & pulsatrix::Featurizer::parameters_module ( )
pure virtual

The trainable parameters, for an optimizer's step() and zero_grad(), and for checkpoints.

Implemented in pulsatrix::BlockSparseFeaturizer, pulsatrix::Crosscoder, pulsatrix::JumpReLUSparseAutoencoder, pulsatrix::SparseAutoencoder, pulsatrix::TopKSparseAutoencoder, and pulsatrix::Transcoder.

◆ predict()

virtual Tensor pulsatrix::Featurizer::predict ( const Tensor &  x)
inlinevirtual

The full output for a batch, (N, output_dim): decode(encode(x)), plus any part that bypasses the codes (a transcoder's skip connection).

Reimplemented in pulsatrix::Transcoder.


The documentation for this class was generated from the following file: