pulsatrix 1.1.0
Loading...
Searching...
No Matches
rms_norm_module.hpp
Go to the documentation of this file.
1
6#pragma once
7
8#include <optional>
9#include <initializer_list>
10#include <vector>
11
12#include "pulsatrix/module.hpp"
13
14namespace pulsatrix {
15
32class RMSNormModule : public Module {
33public:
46 RMSNormModule(int64_t num_features, DeviceBackend* backend, DeviceType device,
47 float eps = 1e-6f);
48
51 RMSNormModule(int64_t num_features, DeviceBackend* backend);
52
61 [[nodiscard]] Tensor backward(const Tensor& grad_output) override;
62
64 [[nodiscard]] OpType op_type() const override { return OpType::Normalization; }
65
67 void set_gamma(std::initializer_list<float> values);
68
70 void set_gamma(const std::vector<float>& values);
71
77 void set_weight_offset(float offset) { weight_offset_ = offset; }
78 [[nodiscard]] float weight_offset() const { return weight_offset_; }
79
80 [[nodiscard]] const Tensor& gamma() const { return gamma_; }
81 [[nodiscard]] const Tensor& gamma_grad() const { return gamma_grad_; }
84 [[nodiscard]] const Tensor& last_rms() const { return last_rms_; }
85
97 [[nodiscard]] Tensor propagate_relevance(const Tensor& relevance_out, const LRPRuleConfig& config) override;
98
99 [[nodiscard]] std::vector<NamedParamRef> named_parameters() override {
100 return {
101 {"weight", {&gamma_, &gamma_grad_}},
102 };
103 }
104
105
107 [[nodiscard]] std::optional<DeviceType> compute_device() const override { return gamma_.device(); }
108
109 void release_activations() override;
110
111protected:
112 [[nodiscard]] Tensor forward_impl(const Tensor& input) override;
113
114private:
115 int64_t num_features_;
116 float weight_offset_ = 0.0f;
117 float eps_;
118 DeviceBackend* backend_;
119 Tensor gamma_;
120 Tensor gamma_grad_;
121 Tensor last_input_;
122 Tensor last_rms_; // (N,) one rms per batch row, on the module's device
123 bool has_forwarded_ = false;
124};
125
126} // namespace pulsatrix
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
Base class for every layer type (LinearModule, Conv2DModule, activations, ...).
Definition module.hpp:78
y_{n,i} = gamma_i * x_{n,i} / rms(x_n), rms(x_n) = sqrt(mean_i(x_{n,i}^2) + eps), computed independen...
Definition rms_norm_module.hpp:32
RMSNormModule(int64_t num_features, DeviceBackend *backend)
On backend's own device (backend->device()), default eps. Previously the device defaulted to Cpu rega...
Tensor propagate_relevance(const Tensor &relevance_out, const LRPRuleConfig &config) override
Identity-rule LRP relevance propagation (AttnLRP, Achtibat et al. 2024).
RMSNormModule(int64_t num_features, DeviceBackend *backend, DeviceType device, float eps=1e-6f)
Constructs an RMSNorm layer with zero-initialized gamma.
void set_gamma(std::initializer_list< float > values)
Overwrites the gamma buffer – test/initialization use only.
Tensor backward(const Tensor &grad_output) override
Computes the gradient w.r.t. this module's input, and accumulates gamma's gradient internally.
Tensor forward_impl(const Tensor &input) override
The actual forward computation. Called by forward() after precondition checks.
void release_activations() override
Frees the tensors forward() kept for backward() and the relevance passes; containers release their ch...
float weight_offset() const
Definition rms_norm_module.hpp:78
std::optional< DeviceType > compute_device() const override
Where this layer computes, so forward() rejects an input on another device (FND-8).
Definition rms_norm_module.hpp:107
const Tensor & last_rms() const
The last forward()'s rms per row, (N,) on the module's device (FEAT-10's frozen normalization scales)...
Definition rms_norm_module.hpp:84
std::vector< NamedParamRef > named_parameters() override
This module's trainable parameters, each with its hierarchical name – the one place a module declares...
Definition rms_norm_module.hpp:99
void set_gamma(const std::vector< float > &values)
Vector overload for runtime-sized sources – see Tensor's own vector ctor.
OpType op_type() const override
Normalization per charter's closed OpType set.
Definition rms_norm_module.hpp:64
const Tensor & gamma() const
Definition rms_norm_module.hpp:80
const Tensor & gamma_grad() const
Definition rms_norm_module.hpp:81
void set_weight_offset(float offset)
Scales by offset + gamma instead of gamma (LLM-9). Gemma stores its RMSNorm weights centered on zero ...
Definition rms_norm_module.hpp:77
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
DeviceType device() const
Which device this tensor's buffer conceptually resides on.
Definition tensor.hpp:122
Abstract base every layer subclasses – NVI forward(), pure-virtual LRP contract.
Definition acquisition_functions.hpp:16
DeviceType
Which physical device a Tensor's buffer resides on.
Definition device_backend.hpp:17
OpType
The op-type tag a Node carries. Charter Part 2 §3: nodes are tagged by a small closed set of op types...
Definition op_type.hpp:19
Configuration for LRP relevance propagation: which rule a module applies and its hyperparameters....
Definition lrp_rule_config.hpp:72