67 [[nodiscard]] int64_t
input_dim()
const override {
return dim_; }
94 void Check(
const Tensor& x,
const char* who)
const;
96 [[nodiscard]] std::vector<float> Codes(
const std::vector<float>& pre)
const;
99 [[nodiscard]]
Tensor BackwardFromCodes(
const std::vector<float>& pre,
const std::vector<float>& g_codes,
double l0_weight);
107 Tensor log_threshold_grad_;
108 std::vector<float> last_pre_;
109 int64_t last_rows_ = 0;
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
virtual DeviceType device() const noexcept=0
Which device this backend's buffers reside on.
A featurizer maps activations x, (N, input_dim), to codes f = encode(x), (N, num_features),...
Definition featurizer.hpp:45
A JumpReLU sparse autoencoder: z = pre · H(pre - θ) with pre = x W_enc + b_enc, and x̂ = z W_dec + b_...
Definition jumprelu_sparse_autoencoder.hpp:49
Module & parameters_module() override
The trainable parameters, for an optimizer's step() and zero_grad(), and for checkpoints.
Definition jumprelu_sparse_autoencoder.hpp:75
FeaturizerLoss loss_and_backward(const Tensor &x, std::vector< float > *codes=nullptr) override
The loss and its gradients; sparsity holds λ * L0.
Tensor forward_impl(const Tensor &input) override
void initialize_bias(const Tensor &x)
Sets b_dec to the mean of x, (N, dim).
LinearModule & encoder()
Definition jumprelu_sparse_autoencoder.hpp:63
int64_t num_features() const override
Definition jumprelu_sparse_autoencoder.hpp:68
void set_training(bool training) override
Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Mod...
Tensor backward(const Tensor &grad_output) override
Backward through the decoder, the active latents and the encoder (the thresholds get the straight-thr...
LinearModule & decoder()
Definition jumprelu_sparse_autoencoder.hpp:64
void set_l0_coefficient(float lambda)
Changes λ, for example to warm it up over the first steps as the paper does.
const JumpReLUSaeOptions & options() const
Definition jumprelu_sparse_autoencoder.hpp:62
OpType op_type() const override
This module's operation-category tag, for ComputationGraph node tagging.
Definition jumprelu_sparse_autoencoder.hpp:83
void release_activations() override
Frees the tensors forward() kept for backward() and the relevance passes; containers release their ch...
std::vector< NamedParamRef > named_parameters() override
This module's trainable parameters, each with its hierarchical name – the one place a module declares...
Tensor encode(const Tensor &x) override
The codes for a batch, (N, num_features).
void normalize_decoder() override
Rescales every feature so its decoder direction has unit L2 norm, scaling its code by the inverse whe...
std::vector< float > decoder_direction(int64_t i) override
Feature i's decoder direction, output_dim values.
Tensor decode(const Tensor &codes) override
The reconstruction from codes, (N, output_dim).
std::optional< DeviceType > compute_device() const override
The device this module computes on, so forward() can reject an input on another device before any ker...
Definition jumprelu_sparse_autoencoder.hpp:85
Tensor propagate_relevance(const Tensor &relevance_out, const LRPRuleConfig &config) override
The decoder's and the encoder's rules, passed through the active latents.
int64_t input_dim() const override
Definition jumprelu_sparse_autoencoder.hpp:67
JumpReLUSparseAutoencoder(int64_t dim, int64_t num_features, DeviceBackend *backend, JumpReLUSaeOptions options={})
std::vector< float > thresholds() const
Each latent's threshold θ.
y = x @ W + b, batched (x is (N, in_features), y is (N, out_features)) – migrated from the original u...
Definition linear_module.hpp:43
Base class for every layer type (LinearModule, Conv2DModule, activations, ...).
Definition module.hpp:78
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
The featurizer family's common interface (FEAT-1): models that rewrite activations as sparse codes ov...
Dense/fully-connected layer – the reference Module implementation.
Definition acquisition_functions.hpp:16
OpType
The op-type tag a Node carries. Charter Part 2 §3: nodes are tagged by a small closed set of op types...
Definition op_type.hpp:19
One batch's loss, split into its parts.
Definition featurizer.hpp:18
Definition jumprelu_sparse_autoencoder.hpp:19
uint64_t seed
Definition jumprelu_sparse_autoencoder.hpp:28
float initial_threshold
Every latent's starting threshold θ (the paper's 0.001).
Definition jumprelu_sparse_autoencoder.hpp:27
float l0_coefficient
λ, the L0 penalty's weight: the loss is mean((x̂ - x)²) + λ * L0, L0 averaged over the batch....
Definition jumprelu_sparse_autoencoder.hpp:23
float bandwidth
ε, the bandwidth of the rectangle kernel that stands in for the step's derivative.
Definition jumprelu_sparse_autoencoder.hpp:25
Configuration for LRP relevance propagation: which rule a module applies and its hyperparameters....
Definition lrp_rule_config.hpp:72