|
| | JumpReLUSparseAutoencoder (int64_t dim, int64_t num_features, DeviceBackend *backend, JumpReLUSaeOptions options={}) |
| |
| void | initialize_bias (const Tensor &x) |
| | Sets b_dec to the mean of x, (N, dim).
|
| |
| void | set_l0_coefficient (float lambda) |
| | Changes λ, for example to warm it up over the first steps as the paper does.
|
| |
| std::vector< float > | thresholds () const |
| | Each latent's threshold θ.
|
| |
| const JumpReLUSaeOptions & | options () const |
| |
| LinearModule & | encoder () |
| |
| LinearModule & | decoder () |
| |
| int64_t | input_dim () const override |
| |
| int64_t | num_features () const override |
| |
| Tensor | encode (const Tensor &x) override |
| | The codes for a batch, (N, num_features).
|
| |
| Tensor | decode (const Tensor &codes) override |
| | The reconstruction from codes, (N, output_dim).
|
| |
| FeaturizerLoss | loss_and_backward (const Tensor &x, std::vector< float > *codes=nullptr) override |
| | The loss and its gradients; sparsity holds λ * L0.
|
| |
| void | normalize_decoder () override |
| | Rescales every feature so its decoder direction has unit L2 norm, scaling its code by the inverse where the encoder allows it, so reconstructions don't change. Training with unit-norm decoders keeps a sparsity penalty from being dodged by shrinking codes and growing directions, and makes code sizes comparable across features.
|
| |
| std::vector< float > | decoder_direction (int64_t i) override |
| | Feature i's decoder direction, output_dim values.
|
| |
| Module & | parameters_module () override |
| | The trainable parameters, for an optimizer's step() and zero_grad(), and for checkpoints.
|
| |
| Tensor | backward (const Tensor &grad_output) override |
| | Backward through the decoder, the active latents and the encoder (the thresholds get the straight-through gradient too).
|
| |
| Tensor | propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override |
| | The decoder's and the encoder's rules, passed through the active latents.
|
| |
| OpType | op_type () const override |
| | This module's operation-category tag, for ComputationGraph node tagging.
|
| |
| std::vector< NamedParamRef > | named_parameters () override |
| | This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).
|
| |
| std::optional< DeviceType > | compute_device () const override |
| | The device this module computes on, so forward() can reject an input on another device before any kernel sees it (roadmap FND-8).
|
| |
| void | set_training (bool training) override |
| | Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
|
| |
| void | release_activations () override |
| | Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
|
| |
| virtual | ~Module ()=default |
| |
| Tensor | forward (const Tensor &input) |
| | Runs this module's forward computation.
|
| |
| std::pair< Tensor, NodeId > | forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd) |
| | Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
|
| |
| virtual bool | supports_lrp_rule (LRPRule rule) const |
| | Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
|
| |
| virtual std::vector< NamedBufferRef > | named_buffers () |
| | This module's buffers: state a checkpoint must save that no optimizer updates.
|
| |
| virtual std::vector< NamedModuleRef > | named_children () |
| | This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
|
| |
| virtual std::vector< ParamRef > | parameters () |
| | This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
|
| |
| void | set_requires_grad (bool requires_grad, const std::string &prefix="") |
| | Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
|
| |
| bool | is_training () const |
| | Whether this module is currently in training mode.
|
| |
| virtual | ~Featurizer ()=default |
| |
| virtual int64_t | output_dim () const |
| | What decode() produces: input_dim() for an autoencoder.
|
| |
| virtual int64_t | block_size () const |
| | Codes come in blocks of this many (FEAT-6): a block-sparse featurizer's feature is a block, active when any of its codes is nonzero, and its codes may be negative. 1 (the default) for featurizers whose features are single directions.
|
| |
| virtual Tensor | predict (const Tensor &x) |
| | The full output for a batch, (N, output_dim): decode(encode(x)), plus any part that bypasses the codes (a transcoder's skip connection).
|
| |
| virtual FeaturizerLoss | loss_and_backward_with_target (const Tensor &x, const Tensor &target, std::vector< float > *codes=nullptr) |
| | The loss for predicting target, (N, output_dim), from x: a transcoder's.
|
| |
A JumpReLU sparse autoencoder: z = pre · H(pre - θ) with pre = x W_enc + b_enc, and x̂ = z W_dec + b_dec. Each latent has its own threshold θ > 0, kept as log θ.
- Training. The step H has no useful derivative, so the paper's straight-through estimators stand in: with K the rectangle kernel (1 on (-1/2, 1/2)),
∂z/∂pre = H(pre - θ), ∂z/∂θ = -(θ/ε) K((pre - θ)/ε) and ∂H/∂θ = -(1/ε) K((pre - θ)/ε). They give θ a gradient from both the reconstruction and the L0 penalty, from inputs within ε/2 of it.
- Sparsity comes from the L0 penalty, not L1, so codes above the threshold aren't shrunk.
- Decoder directions are made unit by normalize_decoder(), which moves each scale into the latent's encoder column, bias and threshold, so codes and reconstructions don't change.
- Initialization: random unit decoder directions, the encoder their transpose,
b_enc = 0, b_dec = 0 (initialize_bias() sets it to the data's mean) and θ = initial_threshold.
A Module from input to reconstruction, with parameters encoder.*, decoder.* and log_threshold.