|
| | Transcoder (int64_t input_dim, int64_t output_dim, int64_t num_features, DeviceBackend *backend, TranscoderOptions options={}) |
| |
| void | initialize_bias (const Tensor &target) |
| | Sets b_dec to the mean of target, (N, output_dim).
|
| |
| int64_t | input_dim () const override |
| |
| int64_t | output_dim () const override |
| | What decode() produces: input_dim() for an autoencoder.
|
| |
| int64_t | num_features () const override |
| |
| Tensor | encode (const Tensor &x) override |
| | The codes for a batch, (N, num_features).
|
| |
| Tensor | decode (const Tensor &codes) override |
| | codes W_dec + b_dec: the prediction without the skip connection.
|
| |
| Tensor | predict (const Tensor &x) override |
| | The full prediction, skip connection included.
|
| |
| FeaturizerLoss | loss_and_backward (const Tensor &x, std::vector< float > *codes=nullptr) override |
| |
| FeaturizerLoss | loss_and_backward_with_target (const Tensor &x, const Tensor &target, std::vector< float > *codes=nullptr) override |
| | The loss and its gradients; reconstruction is the prediction's mean squared error, sparsity the weighted AuxK loss. Counts the batch toward dead-latent tracking.
|
| |
| void | normalize_decoder () override |
| | Rescales every feature so its decoder direction has unit L2 norm, scaling its code by the inverse where the encoder allows it, so reconstructions don't change. Training with unit-norm decoders keeps a sparsity penalty from being dodged by shrinking codes and growing directions, and makes code sizes comparable across features.
|
| |
| std::vector< float > | decoder_direction (int64_t i) override |
| | Feature i's decoder direction, output_dim values.
|
| |
| Module & | parameters_module () override |
| | The trainable parameters, for an optimizer's step() and zero_grad(), and for checkpoints.
|
| |
| std::vector< int64_t > | dead_latents () const |
| |
| const std::vector< int64_t > & | inputs_since_fired () const |
| |
| void | set_inputs_since_fired (const std::vector< int64_t > &since) |
| |
| const TranscoderOptions & | options () const |
| |
| LinearModule & | encoder () |
| |
| LinearModule & | decoder () |
| |
| LinearModule * | skip () |
| | The skip connection; null unless options.skip.
|
| |
| Tensor | backward (const Tensor &grad_output) override |
| | Backward through the latents (and the skip connection), adding to the parameters' gradients.
|
| |
| Tensor | propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override |
| | Relevance of the input: the output's relevance is split between the latents' part and the skip's in proportion to their contributions (epsilon rule), then each passes it back with its own layers' rules.
|
| |
| OpType | op_type () const override |
| | This module's operation-category tag, for ComputationGraph node tagging.
|
| |
| std::vector< NamedParamRef > | named_parameters () override |
| | This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).
|
| |
| std::optional< DeviceType > | compute_device () const override |
| | The device this module computes on, so forward() can reject an input on another device before any kernel sees it (roadmap FND-8).
|
| |
| void | set_training (bool training) override |
| | Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
|
| |
| void | release_activations () override |
| | Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
|
| |
| virtual | ~Module ()=default |
| |
| Tensor | forward (const Tensor &input) |
| | Runs this module's forward computation.
|
| |
| std::pair< Tensor, NodeId > | forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd) |
| | Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
|
| |
| virtual bool | supports_lrp_rule (LRPRule rule) const |
| | Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
|
| |
| virtual std::vector< NamedBufferRef > | named_buffers () |
| | This module's buffers: state a checkpoint must save that no optimizer updates.
|
| |
| virtual std::vector< NamedModuleRef > | named_children () |
| | This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
|
| |
| virtual std::vector< ParamRef > | parameters () |
| | This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
|
| |
| void | set_requires_grad (bool requires_grad, const std::string &prefix="") |
| | Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
|
| |
| bool | is_training () const |
| | Whether this module is currently in training mode.
|
| |
| virtual | ~Featurizer ()=default |
| |
| virtual int64_t | block_size () const |
| | Codes come in blocks of this many (FEAT-6): a block-sparse featurizer's feature is a block, active when any of its codes is nonzero, and its codes may be negative. 1 (the default) for featurizers whose features are single directions.
|
| |
A transcoder: z = ReLU(TopK_k(x W_enc + b_enc)), ŷ = z W_dec + b_dec [+ x W_skip], trained to predict an MLP's output y from its input x by mean((ŷ - y)²) plus AuxK.
- Why. An SAE explains the residual stream; a transcoder explains a computation. Its latents read from the MLP's input and write to its output, so a circuit through the MLP can be traced latent to latent, and Paulo et al. find its latents more interpretable than an SAE's at the same sparsity.
- The skip connection takes the MLP's linear part off the latents. decode() leaves it out (it needs the input); predict() includes it.
- Decoder directions are unit vectors after normalize_decoder(), and the decoder's gradient loses its component along each, as in TopKSparseAutoencoder.
- Initialization: random unit decoder directions, the encoder their transpose (when the input and output sizes match; random otherwise),
b_enc = 0, W_skip = 0, and b_dec 0 until initialize_bias() sets it to the targets' mean.
A Module from input to prediction, with parameters encoder.*, decoder.* and skip.weight. Train it with loss_and_backward_with_target() or TrainFeaturizer(t, x, target, optimizer); loss_and_backward() without a target throws.