pulsatrix 1.1.0
Loading...
Searching...
No Matches
jumprelu_sparse_autoencoder.hpp
Go to the documentation of this file.
1
8#pragma once
9
10#include <cstdint>
11#include <optional>
12#include <vector>
13
16
17namespace pulsatrix {
18
23 float l0_coefficient = 1e-3f;
25 float bandwidth = 1e-3f;
27 float initial_threshold = 1e-3f;
28 uint64_t seed = 0;
29};
30
50public:
54
56 void initialize_bias(const Tensor& x);
59 void set_l0_coefficient(float lambda);
61 [[nodiscard]] std::vector<float> thresholds() const;
62 [[nodiscard]] const JumpReLUSaeOptions& options() const { return options_; }
63 [[nodiscard]] LinearModule& encoder() { return encoder_; }
64 [[nodiscard]] LinearModule& decoder() { return decoder_; }
65
66 // ---- Featurizer ----------------------------------------------------------------------
67 [[nodiscard]] int64_t input_dim() const override { return dim_; }
68 [[nodiscard]] int64_t num_features() const override { return m_; }
69 [[nodiscard]] Tensor encode(const Tensor& x) override;
70 [[nodiscard]] Tensor decode(const Tensor& codes) override;
72 FeaturizerLoss loss_and_backward(const Tensor& x, std::vector<float>* codes = nullptr) override;
73 void normalize_decoder() override;
74 [[nodiscard]] std::vector<float> decoder_direction(int64_t i) override;
75 [[nodiscard]] Module& parameters_module() override { return *this; }
76
77 // ---- Module: input to reconstruction -------------------------------------------------
80 [[nodiscard]] Tensor backward(const Tensor& grad_output) override;
82 [[nodiscard]] Tensor propagate_relevance(const Tensor& relevance_out, const LRPRuleConfig& config) override;
83 [[nodiscard]] OpType op_type() const override { return OpType::Composite; }
84 [[nodiscard]] std::vector<NamedParamRef> named_parameters() override;
85 [[nodiscard]] std::optional<DeviceType> compute_device() const override { return backend_->device(); }
86 void set_training(bool training) override;
87 void release_activations() override;
88
89protected:
91 [[nodiscard]] Tensor forward_impl(const Tensor& input) override;
92
93private:
94 void Check(const Tensor& x, const char* who) const;
96 [[nodiscard]] std::vector<float> Codes(const std::vector<float>& pre) const;
99 [[nodiscard]] Tensor BackwardFromCodes(const std::vector<float>& pre, const std::vector<float>& g_codes, double l0_weight);
100
101 int64_t dim_, m_;
102 DeviceBackend* backend_;
103 JumpReLUSaeOptions options_;
104 LinearModule encoder_;
105 LinearModule decoder_;
106 Tensor log_threshold_;
107 Tensor log_threshold_grad_;
108 std::vector<float> last_pre_;
109 int64_t last_rows_ = 0;
110};
111
112} // namespace pulsatrix
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
virtual DeviceType device() const noexcept=0
Which device this backend's buffers reside on.
A featurizer maps activations x, (N, input_dim), to codes f = encode(x), (N, num_features),...
Definition featurizer.hpp:45
A JumpReLU sparse autoencoder: z = pre · H(pre - θ) with pre = x W_enc + b_enc, and x̂ = z W_dec + b_...
Definition jumprelu_sparse_autoencoder.hpp:49
Module & parameters_module() override
The trainable parameters, for an optimizer's step() and zero_grad(), and for checkpoints.
Definition jumprelu_sparse_autoencoder.hpp:75
FeaturizerLoss loss_and_backward(const Tensor &x, std::vector< float > *codes=nullptr) override
The loss and its gradients; sparsity holds λ * L0.
Tensor forward_impl(const Tensor &input) override
void initialize_bias(const Tensor &x)
Sets b_dec to the mean of x, (N, dim).
LinearModule & encoder()
Definition jumprelu_sparse_autoencoder.hpp:63
int64_t num_features() const override
Definition jumprelu_sparse_autoencoder.hpp:68
void set_training(bool training) override
Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Mod...
Tensor backward(const Tensor &grad_output) override
Backward through the decoder, the active latents and the encoder (the thresholds get the straight-thr...
LinearModule & decoder()
Definition jumprelu_sparse_autoencoder.hpp:64
void set_l0_coefficient(float lambda)
Changes λ, for example to warm it up over the first steps as the paper does.
const JumpReLUSaeOptions & options() const
Definition jumprelu_sparse_autoencoder.hpp:62
OpType op_type() const override
This module's operation-category tag, for ComputationGraph node tagging.
Definition jumprelu_sparse_autoencoder.hpp:83
void release_activations() override
Frees the tensors forward() kept for backward() and the relevance passes; containers release their ch...
std::vector< NamedParamRef > named_parameters() override
This module's trainable parameters, each with its hierarchical name – the one place a module declares...
Tensor encode(const Tensor &x) override
The codes for a batch, (N, num_features).
void normalize_decoder() override
Rescales every feature so its decoder direction has unit L2 norm, scaling its code by the inverse whe...
std::vector< float > decoder_direction(int64_t i) override
Feature i's decoder direction, output_dim values.
Tensor decode(const Tensor &codes) override
The reconstruction from codes, (N, output_dim).
std::optional< DeviceType > compute_device() const override
The device this module computes on, so forward() can reject an input on another device before any ker...
Definition jumprelu_sparse_autoencoder.hpp:85
Tensor propagate_relevance(const Tensor &relevance_out, const LRPRuleConfig &config) override
The decoder's and the encoder's rules, passed through the active latents.
int64_t input_dim() const override
Definition jumprelu_sparse_autoencoder.hpp:67
JumpReLUSparseAutoencoder(int64_t dim, int64_t num_features, DeviceBackend *backend, JumpReLUSaeOptions options={})
std::vector< float > thresholds() const
Each latent's threshold θ.
y = x @ W + b, batched (x is (N, in_features), y is (N, out_features)) – migrated from the original u...
Definition linear_module.hpp:43
Base class for every layer type (LinearModule, Conv2DModule, activations, ...).
Definition module.hpp:78
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
The featurizer family's common interface (FEAT-1): models that rewrite activations as sparse codes ov...
Dense/fully-connected layer – the reference Module implementation.
Definition acquisition_functions.hpp:16
OpType
The op-type tag a Node carries. Charter Part 2 §3: nodes are tagged by a small closed set of op types...
Definition op_type.hpp:19
One batch's loss, split into its parts.
Definition featurizer.hpp:18
Definition jumprelu_sparse_autoencoder.hpp:19
uint64_t seed
Definition jumprelu_sparse_autoencoder.hpp:28
float initial_threshold
Every latent's starting threshold θ (the paper's 0.001).
Definition jumprelu_sparse_autoencoder.hpp:27
float l0_coefficient
λ, the L0 penalty's weight: the loss is mean((x̂ - x)²) + λ * L0, L0 averaged over the batch....
Definition jumprelu_sparse_autoencoder.hpp:23
float bandwidth
ε, the bandwidth of the rectangle kernel that stands in for the step's derivative.
Definition jumprelu_sparse_autoencoder.hpp:25
Configuration for LRP relevance propagation: which rule a module applies and its hyperparameters....
Definition lrp_rule_config.hpp:72