A crosscoder over S sources of the same size d. Its input is the sources side by side, (N, S·d), source s in columns [s·d, (s+1)·d): f = ReLU(Σ_s x^s W_enc^s + b_enc), x̂^s = f W_dec^s + b_dec^s.
More...
|
| | Crosscoder (int64_t num_sources, int64_t source_dim, int64_t num_features, DeviceBackend *backend, CrosscoderOptions options={}) |
| |
| void | initialize_bias (const Tensor &x) |
| | Sets b_dec to the mean of x, (N, S·d).
|
| |
| int64_t | num_sources () const |
| |
| int64_t | source_dim () const |
| |
| int64_t | num_shared () const |
| | With delta: latents [0, num_shared()) are shared, the rest delta latents.
|
| |
| int64_t | input_dim () const override |
| |
| int64_t | num_features () const override |
| |
| Tensor | encode (const Tensor &x) override |
| | The codes: BatchTopK by threshold once one is set, else across this batch.
|
| |
| Tensor | decode (const Tensor &codes) override |
| | The reconstruction from codes, (N, output_dim).
|
| |
| FeaturizerLoss | loss_and_backward (const Tensor &x, std::vector< float > *codes=nullptr) override |
| | The loss and its gradients: reconstruction is the mean squared error over every source, sparsity the weighted L1 penalty or AuxK loss, total adds the delta loss (delta_loss()). Counts the batch toward dead latents and moves the thresholds.
|
| |
| void | normalize_decoder () override |
| | Rescales every feature so its decoder direction has unit L2 norm, scaling its code by the inverse where the encoder allows it, so reconstructions don't change. Training with unit-norm decoders keeps a sparsity penalty from being dodged by shrinking codes and growing directions, and makes code sizes comparable across features.
|
| |
| std::vector< float > | decoder_direction (int64_t i) override |
| | Latent i's decoder in every source, S·d values.
|
| |
| Module & | parameters_module () override |
| | The trainable parameters, for an optimizer's step() and zero_grad(), and for checkpoints.
|
| |
| std::vector< float > | decoder_direction (int64_t i, int64_t s) |
| | Latent i's decoder in source s, d values.
|
| |
| float | delta_loss () const |
| | The last loss_and_backward()'s weighted delta loss (0 without delta).
|
| |
| std::vector< int64_t > | dead_latents () const |
| |
| const std::vector< int64_t > & | inputs_since_fired () const |
| |
| void | set_inputs_since_fired (const std::vector< int64_t > &since) |
| |
| std::vector< float > | thresholds () const |
| | BatchTopK's inference thresholds on f_i · Σ_s |d_i^s| (shared, then delta, with delta); negative until a training batch sets them.
|
| |
| void | set_thresholds (const std::vector< float > &thresholds) |
| |
| const CrosscoderOptions & | options () const |
| |
| LinearModule & | encoder () |
| |
| LinearModule & | decoder () |
| |
| Tensor | backward (const Tensor &grad_output) override |
| | Computes the gradient w.r.t. this module's input, given the gradient w.r.t. its output. Must be called after forward() – uses state cached from that call.
|
| |
| Tensor | propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override |
| | Relevance of every source's input: the decoder's and the encoder's rules, through the selected latents (a fixed gate), as the other featurizers do.
|
| |
| OpType | op_type () const override |
| | This module's operation-category tag, for ComputationGraph node tagging.
|
| |
| std::vector< NamedParamRef > | named_parameters () override |
| | This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).
|
| |
| std::vector< NamedBufferRef > | named_buffers () override |
| | This module's buffers: state a checkpoint must save that no optimizer updates.
|
| |
| std::optional< DeviceType > | compute_device () const override |
| | The device this module computes on, so forward() can reject an input on another device before any kernel sees it (roadmap FND-8).
|
| |
| void | set_training (bool training) override |
| | Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
|
| |
| void | release_activations () override |
| | Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
|
| |
| virtual | ~Module ()=default |
| |
| Tensor | forward (const Tensor &input) |
| | Runs this module's forward computation.
|
| |
| std::pair< Tensor, NodeId > | forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd) |
| | Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
|
| |
| virtual bool | supports_lrp_rule (LRPRule rule) const |
| | Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
|
| |
| virtual std::vector< NamedModuleRef > | named_children () |
| | This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
|
| |
| virtual std::vector< ParamRef > | parameters () |
| | This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
|
| |
| void | set_requires_grad (bool requires_grad, const std::string &prefix="") |
| | Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
|
| |
| bool | is_training () const |
| | Whether this module is currently in training mode.
|
| |
| virtual | ~Featurizer ()=default |
| |
| virtual int64_t | output_dim () const |
| | What decode() produces: input_dim() for an autoencoder.
|
| |
| virtual int64_t | block_size () const |
| | Codes come in blocks of this many (FEAT-6): a block-sparse featurizer's feature is a block, active when any of its codes is nonzero, and its codes may be negative. 1 (the default) for featurizers whose features are single directions.
|
| |
| virtual Tensor | predict (const Tensor &x) |
| | The full output for a batch, (N, output_dim): decode(encode(x)), plus any part that bypasses the codes (a transcoder's skip connection).
|
| |
| virtual FeaturizerLoss | loss_and_backward_with_target (const Tensor &x, const Tensor &target, std::vector< float > *codes=nullptr) |
| | The loss for predicting target, (N, output_dim), from x: a transcoder's.
|
| |
A crosscoder over S sources of the same size d. Its input is the sources side by side, (N, S·d), source s in columns [s·d, (s+1)·d): f = ReLU(Σ_s x^s W_enc^s + b_enc), x̂^s = f W_dec^s + b_dec^s.
- One code, a decoder per source. A latent's decoder norm in each source says how much it writes there: about equal for a feature both models share, near zero in one source for a feature only the other has (CrosscoderLatents(), ClassifyLatent()).
- Loss:
mean((x̂ - x)²) over every source's values, plus the L1 penalty or AuxK, plus the delta loss with delta. Lindsey et al. sum the squared errors instead; the mean keeps the loss on the scale of pulsatrix's other featurizers, and with one source and BatchTopK the crosscoder is a BatchTopK SAE without the encoder's bias subtraction.
- Initialization (Minder et al.): random decoder directions, the same in every source, at decoder_init_norm; the encoder their transpose; zero biases until initialize_bias().
- Decoder norms carry the result, so training doesn't normalize them: train with TrainFeaturizer with
unit_norm_decoder = false, as the references do. normalize_decoder() is still exact: it scales each latent's whole decoder to unit norm and its encoder by the inverse, which changes no reconstruction, no penalty and no selection.
- Activations: normalize each source separately first, so each contributes comparably (Lindsey et al.); for example scale it so its mean squared norm is d.
A Module from input to reconstruction, with parameters encoder.* and decoder.*; BatchTopK's thresholds are a buffer, threshold (one value, two with delta: shared, delta).