|
| | TopKSparseAutoencoder (int64_t dim, int64_t num_features, DeviceBackend *backend, TopKSaeOptions options={}) |
| |
| void | initialize_bias (const Tensor &x) |
| | Sets b_dec to the mean of x, (N, dim), as Gao et al. start it (they use the geometric median; the mean is close for activations).
|
| |
| int64_t | input_dim () const override |
| |
| int64_t | num_features () const override |
| |
| Tensor | encode (const Tensor &x) override |
| | The codes for a batch, (N, num_features).
|
| |
| Tensor | decode (const Tensor &codes) override |
| | The reconstruction from codes, (N, output_dim).
|
| |
| FeaturizerLoss | loss_and_backward (const Tensor &x, std::vector< float > *codes=nullptr) override |
| | The loss and its gradients; sparsity holds the weighted auxiliary loss. Counts the batch toward dead-latent tracking.
|
| |
| void | normalize_decoder () override |
| | Unit decoder rows (Gao et al.: the encoder isn't rescaled).
|
| |
| std::vector< float > | decoder_direction (int64_t i) override |
| | Feature i's decoder direction, output_dim values.
|
| |
| Module & | parameters_module () override |
| | The trainable parameters, for an optimizer's step() and zero_grad(), and for checkpoints.
|
| |
| std::vector< int64_t > | dead_latents () const |
| | Latents dead now: none fired in the last dead_after training inputs.
|
| |
| const std::vector< int64_t > & | inputs_since_fired () const |
| | Training inputs since each latent last fired.
|
| |
| void | set_inputs_since_fired (const std::vector< int64_t > &since) |
| | Restores inputs_since_fired(), for example when resuming training from a checkpoint.
|
| |
| const TopKSaeOptions & | options () const |
| |
| float | threshold () const |
| | BatchTopK's inference threshold; negative until a training batch sets it.
|
| |
| void | set_threshold (float threshold) |
| | Sets the inference threshold (negative: select across the batch).
|
| |
| LinearModule & | encoder () |
| |
| LinearModule & | decoder () |
| |
| Tensor | backward (const Tensor &grad_output) override |
| | Backward from the reconstruction's gradient, through the decoder, the selected latents, the encoder and the bias subtraction, adding to the parameters' gradients.
|
| |
| Tensor | propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override |
| | Relevance of the centered input x - b_dec: the decoder's and the encoder's own rules, passed through the selected latents.
|
| |
| OpType | op_type () const override |
| | This module's operation-category tag, for ComputationGraph node tagging.
|
| |
| std::vector< NamedParamRef > | named_parameters () override |
| | This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).
|
| |
| std::vector< NamedBufferRef > | named_buffers () override |
| | This module's buffers: state a checkpoint must save that no optimizer updates.
|
| |
| std::optional< DeviceType > | compute_device () const override |
| | The device this module computes on, so forward() can reject an input on another device before any kernel sees it (roadmap FND-8).
|
| |
| void | set_training (bool training) override |
| | Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
|
| |
| void | release_activations () override |
| | Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
|
| |
| virtual | ~Module ()=default |
| |
| Tensor | forward (const Tensor &input) |
| | Runs this module's forward computation.
|
| |
| std::pair< Tensor, NodeId > | forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd) |
| | Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
|
| |
| virtual bool | supports_lrp_rule (LRPRule rule) const |
| | Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
|
| |
| virtual std::vector< NamedModuleRef > | named_children () |
| | This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
|
| |
| virtual std::vector< ParamRef > | parameters () |
| | This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
|
| |
| void | set_requires_grad (bool requires_grad, const std::string &prefix="") |
| | Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
|
| |
| bool | is_training () const |
| | Whether this module is currently in training mode.
|
| |
| virtual | ~Featurizer ()=default |
| |
| virtual int64_t | output_dim () const |
| | What decode() produces: input_dim() for an autoencoder.
|
| |
| virtual int64_t | block_size () const |
| | Codes come in blocks of this many (FEAT-6): a block-sparse featurizer's feature is a block, active when any of its codes is nonzero, and its codes may be negative. 1 (the default) for featurizers whose features are single directions.
|
| |
| virtual Tensor | predict (const Tensor &x) |
| | The full output for a batch, (N, output_dim): decode(encode(x)), plus any part that bypasses the codes (a transcoder's skip connection).
|
| |
| virtual FeaturizerLoss | loss_and_backward_with_target (const Tensor &x, const Tensor &target, std::vector< float > *codes=nullptr) |
| | The loss for predicting target, (N, output_dim), from x: a transcoder's.
|
| |
A TopK sparse autoencoder: z = ReLU(TopK_k(W_enc (x - b_dec) + b_enc)), x̂ = z W_dec + b_dec.
- No L1 penalty. k fixes how many latents fire, so codes aren't shrunk toward zero the way an L1 penalty shrinks them (SparseAutoencoder).
- Dead latents. A latent that stops firing gets no gradient and stays dead. The auxiliary loss has the k_aux largest dead latents reconstruct the residual
x - x̂; its normalized squared error |ê - e|² / |e|², weighted by aux_coefficient, gives them a gradient. The training loss is mean((x̂ - x)²) + aux_coefficient * that.
- Decoder directions have unit norm after every normalize_decoder(). Rows are rescaled as Gao et al. do, without moving the scale into the encoder (which would change which latents are selected), and the decoder's gradient loses its component along each direction.
- Initialization (Gao et al.): random unit decoder directions, the encoder their transpose,
b_enc = 0, and b_dec = 0 until initialize_bias() sets it to the data's mean.
- BatchTopK (batch_topk) selects across the batch in training and by threshold() at inference. Before any training batch has set the threshold, encode() selects across the batch it is given.
- Matryoshka (matryoshka_prefixes) adds the prefixes' losses: FeaturizerLoss::total is their sum plus the auxiliary loss, and
reconstruction is the full dictionary's.
It is a Module from input to reconstruction, with parameters named encoder.* and decoder.*; the decoder's bias is b_dec. BatchTopK's threshold is a buffer, threshold, so checkpoints keep it.