y_{n,i} = gamma_i * x_{n,i} / rms(x_n), rms(x_n) = sqrt(mean_i(x_{n,i}^2) + eps), computed independently per batch row n. Batched ((N, num_features)), migrated from the original unbatched (rank-1) scope by campaign_exai_dl_library_batch_dimension_support – no mean-centering, no beta/bias term (RMSNorm's defining simplification vs. LayerNorm).
More...
|
| | RMSNormModule (int64_t num_features, DeviceBackend *backend, DeviceType device, float eps=1e-6f) |
| | Constructs an RMSNorm layer with zero-initialized gamma.
|
| |
| | RMSNormModule (int64_t num_features, DeviceBackend *backend) |
| | On backend's own device (backend->device()), default eps. Previously the device defaulted to Cpu regardless of backend (GPU-native-kernels Mission 0).
|
| |
| Tensor | backward (const Tensor &grad_output) override |
| | Computes the gradient w.r.t. this module's input, and accumulates gamma's gradient internally.
|
| |
| OpType | op_type () const override |
| | Normalization per charter's closed OpType set.
|
| |
| void | set_gamma (std::initializer_list< float > values) |
| | Overwrites the gamma buffer – test/initialization use only.
|
| |
| void | set_gamma (const std::vector< float > &values) |
| | Vector overload for runtime-sized sources – see Tensor's own vector ctor.
|
| |
| void | set_weight_offset (float offset) |
| | Scales by offset + gamma instead of gamma (LLM-9). Gemma stores its RMSNorm weights centered on zero and multiplies by 1 + w; with an offset of 1 the parameter stays the checkpoint's w, so loading and saving don't change it. The default is 0.
|
| |
| float | weight_offset () const |
| |
| const Tensor & | gamma () const |
| |
| const Tensor & | gamma_grad () const |
| |
| const Tensor & | last_rms () const |
| | The last forward()'s rms per row, (N,) on the module's device (FEAT-10's frozen normalization scales).
|
| |
| Tensor | propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override |
| | Identity-rule LRP relevance propagation (AttnLRP, Achtibat et al. 2024).
|
| |
| std::vector< NamedParamRef > | named_parameters () override |
| | This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).
|
| |
| std::optional< DeviceType > | compute_device () const override |
| | Where this layer computes, so forward() rejects an input on another device (FND-8).
|
| |
| void | release_activations () override |
| | Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
|
| |
| virtual | ~Module ()=default |
| |
| Tensor | forward (const Tensor &input) |
| | Runs this module's forward computation.
|
| |
| std::pair< Tensor, NodeId > | forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd) |
| | Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
|
| |
| virtual bool | supports_lrp_rule (LRPRule rule) const |
| | Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
|
| |
| virtual std::vector< NamedBufferRef > | named_buffers () |
| | This module's buffers: state a checkpoint must save that no optimizer updates.
|
| |
| virtual std::vector< NamedModuleRef > | named_children () |
| | This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
|
| |
| virtual std::vector< ParamRef > | parameters () |
| | This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
|
| |
| void | set_requires_grad (bool requires_grad, const std::string &prefix="") |
| | Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
|
| |
| virtual void | set_training (bool training) |
| | Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
|
| |
| bool | is_training () const |
| | Whether this module is currently in training mode.
|
| |
y_{n,i} = gamma_i * x_{n,i} / rms(x_n), rms(x_n) = sqrt(mean_i(x_{n,i}^2) + eps), computed independently per batch row n. Batched ((N, num_features)), migrated from the original unbatched (rank-1) scope by campaign_exai_dl_library_batch_dimension_support – no mean-centering, no beta/bias term (RMSNorm's defining simplification vs. LayerNorm).
- Note
- propagate_relevance is an identity pass-through, cited to AttnLRP's normalization-layer treatment (Achtibat et al. 2024, already named in this project's charter for Attention/Transformer work) – the same pattern this codebase already uses for ReluModule/FlattenModule's parameterless/ elementwise-treated operations, not a new pattern invented for this module. backward() is the real, undetached training gradient – the identity-rule simplification applies only to relevance redistribution, never to the actual gradient used for parameter updates (mirrors LinearModule's own independent backward()/propagate_relevance() computations from the same cached forward state).