A block-sparse featurizer: G blocks of b directions in a d-dimensional space. Codes are (N, G·b), block g's in columns [g·b, (g+1)·b), signed; the reconstruction is x̂ = z D, D's rows the atoms. A block is active when its codes aren't all zero, and its norm says how strongly; its coordinates say where on the concept's subspace the input is.
More...
|
| | BlockSparseFeaturizer (int64_t dim, int64_t num_blocks, DeviceBackend *backend, BlockSparseOptions options={}) |
| |
| int64_t | num_blocks () const |
| |
| const BlockSparseOptions & | options () const |
| |
| std::vector< float > | block_frame (int64_t g) |
| | Block g's frame, b x d row-major (its b atoms).
|
| |
| std::vector< float > | gamma () const |
| | The Grassmannian γ per block (empty for the other variants).
|
| |
| std::vector< float > | thresholds () const |
| | GroupLasso's thresholds θ per block, and the current λ.
|
| |
| float | l0_multiplier () const |
| |
| LinearModule & | encoder () |
| |
| LinearModule & | decoder () |
| |
| int64_t | input_dim () const override |
| |
| int64_t | num_features () const override |
| | The codes per input, num_blocks · block_size.
|
| |
| int64_t | block_size () const override |
| | Codes come in blocks of this many (FEAT-6): a block-sparse featurizer's feature is a block, active when any of its codes is nonzero, and its codes may be negative. 1 (the default) for featurizers whose features are single directions.
|
| |
| Tensor | encode (const Tensor &x) override |
| | The codes for a batch, (N, num_features).
|
| |
| Tensor | decode (const Tensor &codes) override |
| | The reconstruction from codes, (N, output_dim).
|
| |
| FeaturizerLoss | loss_and_backward (const Tensor &x, std::vector< float > *codes=nullptr) override |
| | Computes the training loss on x and adds its gradients to the parameters' (zero them first). When codes is given, it receives the batch's codes, row-major (N, num_features), for metrics.
|
| |
| void | normalize_decoder () override |
| | Rescales every feature so its decoder direction has unit L2 norm, scaling its code by the inverse where the encoder allows it, so reconstructions don't change. Training with unit-norm decoders keeps a sparsity penalty from being dodged by shrinking codes and growing directions, and makes code sizes comparable across features.
|
| |
| std::vector< float > | decoder_direction (int64_t i) override |
| | Atom i (block i / b, direction i % b), d values.
|
| |
| Module & | parameters_module () override |
| | The trainable parameters, for an optimizer's step() and zero_grad(), and for checkpoints.
|
| |
| Tensor | backward (const Tensor &grad_output) override |
| | Backward with the selection held fixed.
|
| |
| Tensor | propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override |
| | Computes this module's contribution to LRP relevance propagation.
|
| |
| OpType | op_type () const override |
| | This module's operation-category tag, for ComputationGraph node tagging.
|
| |
| std::vector< NamedParamRef > | named_parameters () override |
| | This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).
|
| |
| std::vector< NamedBufferRef > | named_buffers () override |
| | This module's buffers: state a checkpoint must save that no optimizer updates.
|
| |
| std::optional< DeviceType > | compute_device () const override |
| | The device this module computes on, so forward() can reject an input on another device before any kernel sees it (roadmap FND-8).
|
| |
| void | set_training (bool training) override |
| | Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
|
| |
| void | release_activations () override |
| | Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
|
| |
| virtual | ~Module ()=default |
| |
| Tensor | forward (const Tensor &input) |
| | Runs this module's forward computation.
|
| |
| std::pair< Tensor, NodeId > | forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd) |
| | Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
|
| |
| virtual bool | supports_lrp_rule (LRPRule rule) const |
| | Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
|
| |
| virtual std::vector< NamedModuleRef > | named_children () |
| | This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
|
| |
| virtual std::vector< ParamRef > | parameters () |
| | This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
|
| |
| void | set_requires_grad (bool requires_grad, const std::string &prefix="") |
| | Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
|
| |
| bool | is_training () const |
| | Whether this module is currently in training mode.
|
| |
| virtual | ~Featurizer ()=default |
| |
| virtual int64_t | output_dim () const |
| | What decode() produces: input_dim() for an autoencoder.
|
| |
| virtual Tensor | predict (const Tensor &x) |
| | The full output for a batch, (N, output_dim): decode(encode(x)), plus any part that bypasses the codes (a transcoder's skip connection).
|
| |
| virtual FeaturizerLoss | loss_and_backward_with_target (const Tensor &x, const Tensor &target, std::vector< float > *codes=nullptr) |
| | The loss for predicting target, (N, output_dim), from x: a transcoder's.
|
| |
A block-sparse featurizer: G blocks of b directions in a d-dimensional space. Codes are (N, G·b), block g's in columns [g·b, (g+1)·b), signed; the reconstruction is x̂ = z D, D's rows the atoms. A block is active when its codes aren't all zero, and its norm says how strongly; its coordinates say where on the concept's subspace the input is.
- No decoder bias, as in the paper: center the data first (and scale it so the mean squared norm is d, the reference code's convention).
- Training minimizes
mean((x̂ - x)²), plus λ · L0 for GroupLasso (sparsity). Only active blocks pass gradient. GroupLasso's gate has the reference code's straight-through estimator: a Gaussian kernel K at ‖a_g‖ - θ_g sends +K to the block norm and -K to θ_g, from both the reconstruction and the penalty. Its θ starts on the first training batch at the norm quantile that makes target_l0 blocks fire.
- normalize_decoder() gives Vanilla and GroupLasso atoms unit norm, and restores the Grassmannian frames' orthonormality by QR (Gram–Schmidt in row order), as Appendix D of the paper does; the reference code orthonormalizes by QR inside every forward pass instead.
- Relevance (LRP). The selection is a fixed gate: relevance reaches the selected blocks' codes through the decoder's rule and goes back through the encoder's (the Grassmannian γ, a constant gain per block, passes it unchanged). This is the block top-k rule the roadmap asked for.
A Module from input to reconstruction. Parameters: encoder.* and decoder.weight (Vanilla, GroupLasso); decoder.weight (the frames) and gamma (Grassmannian); and threshold_raw (GroupLasso). GroupLasso's bandwidth, λ and initialization flag are buffers. The tournament's wins and verdicts are not saved.