88 [[nodiscard]] int64_t
input_dim()
const override {
return dim_; }
137 std::vector<float> pre;
138 std::vector<float> codes;
142 Pass Encode(
const Tensor& x,
bool training);
143 void Check(
const Tensor& x,
const char* who)
const;
147 void ProjectDecoderGradient();
154 std::vector<int64_t> since_fired_;
156 std::vector<float> last_codes_;
157 int64_t last_rows_ = 0;
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
virtual DeviceType device() const noexcept=0
Which device this backend's buffers reside on.
A featurizer maps activations x, (N, input_dim), to codes f = encode(x), (N, num_features),...
Definition featurizer.hpp:45
y = x @ W + b, batched (x is (N, in_features), y is (N, out_features)) – migrated from the original u...
Definition linear_module.hpp:43
Base class for every layer type (LinearModule, Conv2DModule, activations, ...).
Definition module.hpp:78
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
A TopK sparse autoencoder: z = ReLU(TopK_k(W_enc (x - b_dec) + b_enc)), x̂ = z W_dec + b_dec.
Definition topk_sparse_autoencoder.hpp:76
Tensor encode(const Tensor &x) override
The codes for a batch, (N, num_features).
std::optional< DeviceType > compute_device() const override
The device this module computes on, so forward() can reject an input on another device before any ker...
Definition topk_sparse_autoencoder.hpp:125
const TopKSaeOptions & options() const
Definition topk_sparse_autoencoder.hpp:107
Tensor decode(const Tensor &codes) override
The reconstruction from codes, (N, output_dim).
LinearModule & encoder()
Definition topk_sparse_autoencoder.hpp:112
void set_training(bool training) override
Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Mod...
void normalize_decoder() override
Unit decoder rows (Gao et al.: the encoder isn't rescaled).
int64_t input_dim() const override
Definition topk_sparse_autoencoder.hpp:88
std::vector< float > decoder_direction(int64_t i) override
Feature i's decoder direction, output_dim values.
Module & parameters_module() override
The trainable parameters, for an optimizer's step() and zero_grad(), and for checkpoints.
Definition topk_sparse_autoencoder.hpp:98
void release_activations() override
Frees the tensors forward() kept for backward() and the relevance passes; containers release their ch...
int64_t num_features() const override
Definition topk_sparse_autoencoder.hpp:89
std::vector< NamedBufferRef > named_buffers() override
This module's buffers: state a checkpoint must save that no optimizer updates.
void set_inputs_since_fired(const std::vector< int64_t > &since)
Restores inputs_since_fired(), for example when resuming training from a checkpoint.
const std::vector< int64_t > & inputs_since_fired() const
Training inputs since each latent last fired.
Definition topk_sparse_autoencoder.hpp:103
Tensor propagate_relevance(const Tensor &relevance_out, const LRPRuleConfig &config) override
Relevance of the centered input x - b_dec: the decoder's and the encoder's own rules,...
Tensor backward(const Tensor &grad_output) override
Backward from the reconstruction's gradient, through the decoder, the selected latents,...
void set_threshold(float threshold)
Sets the inference threshold (negative: select across the batch).
std::vector< NamedParamRef > named_parameters() override
This module's trainable parameters, each with its hierarchical name – the one place a module declares...
LinearModule & decoder()
Definition topk_sparse_autoencoder.hpp:113
Tensor forward_impl(const Tensor &input) override
The reconstruction.
FeaturizerLoss loss_and_backward(const Tensor &x, std::vector< float > *codes=nullptr) override
The loss and its gradients; sparsity holds the weighted auxiliary loss. Counts the batch toward dead-...
void initialize_bias(const Tensor &x)
Sets b_dec to the mean of x, (N, dim), as Gao et al. start it (they use the geometric median; the mea...
std::vector< int64_t > dead_latents() const
Latents dead now: none fired in the last dead_after training inputs.
float threshold() const
BatchTopK's inference threshold; negative until a training batch sets it.
TopKSparseAutoencoder(int64_t dim, int64_t num_features, DeviceBackend *backend, TopKSaeOptions options={})
OpType op_type() const override
This module's operation-category tag, for ComputationGraph node tagging.
Definition topk_sparse_autoencoder.hpp:122
The featurizer family's common interface (FEAT-1): models that rewrite activations as sparse codes ov...
Dense/fully-connected layer – the reference Module implementation.
Definition acquisition_functions.hpp:16
OpType
The op-type tag a Node carries. Charter Part 2 §3: nodes are tagged by a small closed set of op types...
Definition op_type.hpp:19
One batch's loss, split into its parts.
Definition featurizer.hpp:18
Configuration for LRP relevance propagation: which rule a module applies and its hyperparameters....
Definition lrp_rule_config.hpp:72
Definition topk_sparse_autoencoder.hpp:20
int64_t dead_after
A latent is dead once it hasn't fired in this many training inputs (Gao et al. use 10 million tokens)...
Definition topk_sparse_autoencoder.hpp:30
uint64_t seed
Definition topk_sparse_autoencoder.hpp:31
float threshold_decay
The threshold's moving-average decay, per training batch.
Definition topk_sparse_autoencoder.hpp:40
bool batch_topk
BatchTopK: in training, keep the batch's N * k largest activations wherever they fall,...
Definition topk_sparse_autoencoder.hpp:38
float aux_coefficient
The auxiliary loss's weight, 1/32 in Gao et al.; 0 turns it off.
Definition topk_sparse_autoencoder.hpp:27
int64_t k
Latents kept per input: L0 is exactly this (or less, where fewer are positive).
Definition topk_sparse_autoencoder.hpp:22
std::vector< int64_t > matryoshka_prefixes
Matryoshka: nested dictionary sizes, increasing (for example {m/16, m/4, m}); the full size is added ...
Definition topk_sparse_autoencoder.hpp:47
int64_t k_aux
The auxiliary loss's latents: the k_aux largest among the dead ones. 0 means half the input dimension...
Definition topk_sparse_autoencoder.hpp:25