|
| | SparseAutoencoder (int64_t dim, int64_t hidden_dim, float l1_lambda, DeviceBackend *backend) |
| | Constructs a sparse autoencoder over activations of a given dimension.
|
| |
| | SparseAutoencoder (int64_t dim, int64_t hidden_dim, float l1_lambda, DeviceBackend *backend, unsigned seed) |
| |
| template<typename OptimizerT > |
| float | train_step (const Tensor &input_batch, OptimizerT &optimizer) |
| | Runs one training step: forward, MSE reconstruction loss, backward with the L1 penalty gradient injected at the hidden layer, one optimizer update of the encoder's and decoder's parameters.
|
| |
| int64_t | input_dim () const override |
| |
| int64_t | num_features () const override |
| |
| Tensor | encode (const Tensor &x) override |
| | The ReLU codes, (N, hidden_dim).
|
| |
| Tensor | decode (const Tensor &codes) override |
| | The reconstruction from codes, (N, dim).
|
| |
| FeaturizerLoss | loss_and_backward (const Tensor &input_batch, std::vector< float > *codes=nullptr) override |
| | Mean squared reconstruction error plus l1_lambda * mean_over_inputs(sum_i f_i), with both gradients added to the parameters'. See train_step() for the L1 term's gradient.
|
| |
| void | normalize_decoder () override |
| | Unit-norm decoder rows, the norm moved into the encoder's row and bias: the ReLU passes a positive scale through, so reconstructions don't change. A feature whose direction is zero is left alone.
|
| |
| std::vector< float > | decoder_direction (int64_t i) override |
| | Feature i's decoder direction, output_dim values.
|
| |
| Module & | parameters_module () override |
| | The trainable parameters, for an optimizer's step() and zero_grad(), and for checkpoints.
|
| |
| bool | unit_norm_decoder () const |
| | Whether train_step() keeps decoder directions at unit norm (the default).
|
| |
| void | set_unit_norm_decoder (bool on) |
| |
| Tensor | backward (const Tensor &grad_output) override |
| | Backward from the reconstruction's gradient to the input's, through the decoder, the ReLU and the encoder, adding to their parameters' gradients.
|
| |
| Tensor | propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override |
| | Each layer's own rule, from the reconstruction back to the input.
|
| |
| OpType | op_type () const override |
| | This module's operation-category tag, for ComputationGraph node tagging.
|
| |
| std::vector< NamedParamRef > | named_parameters () override |
| | This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).
|
| |
| std::optional< DeviceType > | compute_device () const override |
| | The device this module computes on, so forward() can reject an input on another device before any kernel sees it (roadmap FND-8).
|
| |
| void | set_training (bool training) override |
| | Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
|
| |
| void | release_activations () override |
| | Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
|
| |
| Tensor | reconstruct (const Tensor &input_batch) const |
| | The SAE's reconstruction of a batch – the encoder->ReLU->decoder forward path, forward-only: no loss, no backward, no parameter update.
|
| |
| float | reconstruction_error (const Tensor &input_batch) const |
| | Mean squared reconstruction error over the batch – forward-only, no backward, no parameter update.
|
| |
| float | mean_hidden_activation (const Tensor &input_batch) const |
| | Mean hidden (post-ReLU) activation over the batch – the sparsity metric.
|
| |
| int64_t | dim () const |
| | Width of the activation vectors this SAE reconstructs.
|
| |
| int64_t | hidden_dim () const |
| | Width of the sparse hidden basis.
|
| |
| float | l1_lambda () const |
| | Coefficient of the L1 penalty on the hidden activation.
|
| |
| LinearModule & | encoder () |
| | The encoder – inspection (its columns are the learned feature directions, which is the whole point of an SAE) and test-time weight injection.
|
| |
| const LinearModule & | encoder () const |
| | Const overload of encoder().
|
| |
| LinearModule & | decoder () |
| | The decoder – same rationale as encoder().
|
| |
| const LinearModule & | decoder () const |
| | Const overload of decoder().
|
| |
| virtual | ~Module ()=default |
| |
| Tensor | forward (const Tensor &input) |
| | Runs this module's forward computation.
|
| |
| std::pair< Tensor, NodeId > | forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd) |
| | Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
|
| |
| virtual bool | supports_lrp_rule (LRPRule rule) const |
| | Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
|
| |
| virtual std::vector< NamedBufferRef > | named_buffers () |
| | This module's buffers: state a checkpoint must save that no optimizer updates.
|
| |
| virtual std::vector< NamedModuleRef > | named_children () |
| | This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
|
| |
| virtual std::vector< ParamRef > | parameters () |
| | This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
|
| |
| void | set_requires_grad (bool requires_grad, const std::string &prefix="") |
| | Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
|
| |
| bool | is_training () const |
| | Whether this module is currently in training mode.
|
| |
| virtual | ~Featurizer ()=default |
| |
| virtual int64_t | output_dim () const |
| | What decode() produces: input_dim() for an autoencoder.
|
| |
| virtual int64_t | block_size () const |
| | Codes come in blocks of this many (FEAT-6): a block-sparse featurizer's feature is a block, active when any of its codes is nonzero, and its codes may be negative. 1 (the default) for featurizers whose features are single directions.
|
| |
| virtual Tensor | predict (const Tensor &x) |
| | The full output for a batch, (N, output_dim): decode(encode(x)), plus any part that bypasses the codes (a transcoder's skip connection).
|
| |
| virtual FeaturizerLoss | loss_and_backward_with_target (const Tensor &x, const Tensor &target, std::vector< float > *codes=nullptr) |
| | The loss for predicting target, (N, output_dim), from x: a transcoder's.
|
| |
A sparse autoencoder (SAE): LinearModule(dim, hidden_dim) -> ReluModule -> LinearModule(hidden_dim, dim), trained with MSELoss to reconstruct its own input while an L1 penalty on the hidden ReLU activation pushes most hidden units to zero on any given example.
hidden_dim is normally chosen larger than dim – an overcomplete basis, more directions than the activation space being decomposed, which is what distinguishes an SAE from a compressing autoencoder. The activations come from an ActivationSnapshot (this campaign's Phase 1 output) in production use, but nothing here depends on that: an SAE consumes a plain (N, dim) batch, so it is equally usable against hand-built synthetic data – which is exactly what the paired penalty/no-penalty control in tests/sparse_autoencoder_test.cpp needs.
- Note
- Bounded claim (campaign Decision Point 3): this class measures and reports reconstruction fidelity (reconstruction_error) and sparsity (mean_hidden_activation) as numbers. It does not establish that the hidden directions it learns are semantically meaningful or causally compositional features – that is a live debate in the field, and Phase 4 (activation patching) evidence is the minimum prerequisite for even arguing it. Do not let a low reconstruction error be read as a claim about feature semantics.
-
Mirrors LinearProbe/XorNetwork's established shape rather than inventing a training abstraction: the constructor takes a backend,
train_step() drives one loss/backward/optimizer-step cycle, and the epoch loop is the caller's. The SAE deliberately does not own its optimizer – same division of responsibility LinearProbe::train_step already uses, which is what lets a caller choose SGD vs. Adam and keep one optimizer's state across a whole training run.
-
train_step is a template on the optimizer type rather than taking an Optimizer&, for exactly the reason LinearProbe documents: this codebase has no Optimizer base class. SGDOptimizer, AdamOptimizer and AdamWOptimizer share the step(Module&) / zero_grad(Module&) shape (a compile-time, duck-typed contract), so the template accepts any of them.
-
Small seeded random weight init, not zero init – and here, unlike LinearProbe, zero init is not merely degenerate but dead: with an all-zero decoder weight the gradient reaching the hidden layer is identically zero, so the encoder never receives any gradient and the autoencoder cannot leave the origin.
-
Featurizer (FEAT-1). It implements the Featurizer interface: encode() gives the ReLU codes, decode() the reconstruction, and TrainFeaturizer() trains it like any other featurizer. It is also a Module from input to reconstruction, with backward() and LRP, and parameters named
encoder.* and decoder.*.
-
Unit-norm decoders. By default every feature's decoder direction (a row of the decoder's weight) has unit L2 norm: at construction and after every train_step(). The rescaling moves the norm into the encoder's row and bias, which a ReLU passes through unchanged, so reconstructions don't move. Without it an L1 penalty can be dodged by shrinking codes and growing directions. set_unit_norm_decoder(false) turns it off.
-
hidden_dim > dim is not enforced. An undercomplete or square autoencoder is still a well-formed object with well-defined training behavior; rejecting it would turn a modeling choice into a hard error for no correctness gain. Explicit decision (mission exit gate asked for it to be made, not assumed), pinned by ConstructorAcceptsAnUndercompleteHiddenDimension.