pulsatrix 1.1.0
Loading...
Searching...
No Matches
topk_sparse_autoencoder.hpp
Go to the documentation of this file.
1
9#pragma once
10
11#include <cstdint>
12#include <optional>
13#include <vector>
14
17
18namespace pulsatrix {
19
22 int64_t k = 32;
25 int64_t k_aux = 0;
27 float aux_coefficient = 1.0f / 32.0f;
30 int64_t dead_after = 10'000'000;
31 uint64_t seed = 0;
38 bool batch_topk = false;
40 float threshold_decay = 0.99f;
47 std::vector<int64_t> matryoshka_prefixes;
48};
49
76class TopKSparseAutoencoder : public Module, public Featurizer {
77public:
82
85 void initialize_bias(const Tensor& x);
86
87 // ---- Featurizer ----------------------------------------------------------------------
88 [[nodiscard]] int64_t input_dim() const override { return dim_; }
89 [[nodiscard]] int64_t num_features() const override { return m_; }
90 [[nodiscard]] Tensor encode(const Tensor& x) override;
91 [[nodiscard]] Tensor decode(const Tensor& codes) override;
94 FeaturizerLoss loss_and_backward(const Tensor& x, std::vector<float>* codes = nullptr) override;
96 void normalize_decoder() override;
97 [[nodiscard]] std::vector<float> decoder_direction(int64_t i) override;
98 [[nodiscard]] Module& parameters_module() override { return *this; }
99
101 [[nodiscard]] std::vector<int64_t> dead_latents() const;
103 [[nodiscard]] const std::vector<int64_t>& inputs_since_fired() const { return since_fired_; }
106 void set_inputs_since_fired(const std::vector<int64_t>& since);
107 [[nodiscard]] const TopKSaeOptions& options() const { return options_; }
109 [[nodiscard]] float threshold() const;
112 [[nodiscard]] LinearModule& encoder() { return encoder_; }
113 [[nodiscard]] LinearModule& decoder() { return decoder_; }
114
115 // ---- Module: input to reconstruction -------------------------------------------------
118 [[nodiscard]] Tensor backward(const Tensor& grad_output) override;
121 [[nodiscard]] Tensor propagate_relevance(const Tensor& relevance_out, const LRPRuleConfig& config) override;
122 [[nodiscard]] OpType op_type() const override { return OpType::Composite; }
123 [[nodiscard]] std::vector<NamedParamRef> named_parameters() override;
124 [[nodiscard]] std::vector<NamedBufferRef> named_buffers() override;
125 [[nodiscard]] std::optional<DeviceType> compute_device() const override { return backend_->device(); }
126 void set_training(bool training) override;
127 void release_activations() override;
128
129protected:
131 [[nodiscard]] Tensor forward_impl(const Tensor& input) override;
132
133private:
136 struct Pass {
137 std::vector<float> pre;
138 std::vector<float> codes;
139 float min_kept = 0;
140 };
142 Pass Encode(const Tensor& x, bool training);
143 void Check(const Tensor& x, const char* who) const;
145 Tensor Centered(const Tensor& x);
147 void ProjectDecoderGradient();
148
149 int64_t dim_, m_;
150 DeviceBackend* backend_;
151 TopKSaeOptions options_;
152 LinearModule encoder_;
153 LinearModule decoder_;
154 std::vector<int64_t> since_fired_;
155 Tensor threshold_;
156 std::vector<float> last_codes_;
157 int64_t last_rows_ = 0;
158};
159
160} // namespace pulsatrix
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
virtual DeviceType device() const noexcept=0
Which device this backend's buffers reside on.
A featurizer maps activations x, (N, input_dim), to codes f = encode(x), (N, num_features),...
Definition featurizer.hpp:45
y = x @ W + b, batched (x is (N, in_features), y is (N, out_features)) – migrated from the original u...
Definition linear_module.hpp:43
Base class for every layer type (LinearModule, Conv2DModule, activations, ...).
Definition module.hpp:78
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
A TopK sparse autoencoder: z = ReLU(TopK_k(W_enc (x - b_dec) + b_enc)), x̂ = z W_dec + b_dec.
Definition topk_sparse_autoencoder.hpp:76
Tensor encode(const Tensor &x) override
The codes for a batch, (N, num_features).
std::optional< DeviceType > compute_device() const override
The device this module computes on, so forward() can reject an input on another device before any ker...
Definition topk_sparse_autoencoder.hpp:125
const TopKSaeOptions & options() const
Definition topk_sparse_autoencoder.hpp:107
Tensor decode(const Tensor &codes) override
The reconstruction from codes, (N, output_dim).
LinearModule & encoder()
Definition topk_sparse_autoencoder.hpp:112
void set_training(bool training) override
Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Mod...
void normalize_decoder() override
Unit decoder rows (Gao et al.: the encoder isn't rescaled).
int64_t input_dim() const override
Definition topk_sparse_autoencoder.hpp:88
std::vector< float > decoder_direction(int64_t i) override
Feature i's decoder direction, output_dim values.
Module & parameters_module() override
The trainable parameters, for an optimizer's step() and zero_grad(), and for checkpoints.
Definition topk_sparse_autoencoder.hpp:98
void release_activations() override
Frees the tensors forward() kept for backward() and the relevance passes; containers release their ch...
int64_t num_features() const override
Definition topk_sparse_autoencoder.hpp:89
std::vector< NamedBufferRef > named_buffers() override
This module's buffers: state a checkpoint must save that no optimizer updates.
void set_inputs_since_fired(const std::vector< int64_t > &since)
Restores inputs_since_fired(), for example when resuming training from a checkpoint.
const std::vector< int64_t > & inputs_since_fired() const
Training inputs since each latent last fired.
Definition topk_sparse_autoencoder.hpp:103
Tensor propagate_relevance(const Tensor &relevance_out, const LRPRuleConfig &config) override
Relevance of the centered input x - b_dec: the decoder's and the encoder's own rules,...
Tensor backward(const Tensor &grad_output) override
Backward from the reconstruction's gradient, through the decoder, the selected latents,...
void set_threshold(float threshold)
Sets the inference threshold (negative: select across the batch).
std::vector< NamedParamRef > named_parameters() override
This module's trainable parameters, each with its hierarchical name – the one place a module declares...
LinearModule & decoder()
Definition topk_sparse_autoencoder.hpp:113
Tensor forward_impl(const Tensor &input) override
The reconstruction.
FeaturizerLoss loss_and_backward(const Tensor &x, std::vector< float > *codes=nullptr) override
The loss and its gradients; sparsity holds the weighted auxiliary loss. Counts the batch toward dead-...
void initialize_bias(const Tensor &x)
Sets b_dec to the mean of x, (N, dim), as Gao et al. start it (they use the geometric median; the mea...
std::vector< int64_t > dead_latents() const
Latents dead now: none fired in the last dead_after training inputs.
float threshold() const
BatchTopK's inference threshold; negative until a training batch sets it.
TopKSparseAutoencoder(int64_t dim, int64_t num_features, DeviceBackend *backend, TopKSaeOptions options={})
OpType op_type() const override
This module's operation-category tag, for ComputationGraph node tagging.
Definition topk_sparse_autoencoder.hpp:122
The featurizer family's common interface (FEAT-1): models that rewrite activations as sparse codes ov...
Dense/fully-connected layer – the reference Module implementation.
Definition acquisition_functions.hpp:16
OpType
The op-type tag a Node carries. Charter Part 2 §3: nodes are tagged by a small closed set of op types...
Definition op_type.hpp:19
One batch's loss, split into its parts.
Definition featurizer.hpp:18
Configuration for LRP relevance propagation: which rule a module applies and its hyperparameters....
Definition lrp_rule_config.hpp:72
Definition topk_sparse_autoencoder.hpp:20
int64_t dead_after
A latent is dead once it hasn't fired in this many training inputs (Gao et al. use 10 million tokens)...
Definition topk_sparse_autoencoder.hpp:30
uint64_t seed
Definition topk_sparse_autoencoder.hpp:31
float threshold_decay
The threshold's moving-average decay, per training batch.
Definition topk_sparse_autoencoder.hpp:40
bool batch_topk
BatchTopK: in training, keep the batch's N * k largest activations wherever they fall,...
Definition topk_sparse_autoencoder.hpp:38
float aux_coefficient
The auxiliary loss's weight, 1/32 in Gao et al.; 0 turns it off.
Definition topk_sparse_autoencoder.hpp:27
int64_t k
Latents kept per input: L0 is exactly this (or less, where fewer are positive).
Definition topk_sparse_autoencoder.hpp:22
std::vector< int64_t > matryoshka_prefixes
Matryoshka: nested dictionary sizes, increasing (for example {m/16, m/4, m}); the full size is added ...
Definition topk_sparse_autoencoder.hpp:47
int64_t k_aux
The auxiliary loss's latents: the k_aux largest among the dead ones. 0 means half the input dimension...
Definition topk_sparse_autoencoder.hpp:25