RMS normalization layer (Zhang & Sennrich, 2019) – this codebase's first normalization Module, establishing the Tier 1 pattern for GroupNorm/LayerNorm. More...

#include <optional>
#include <initializer_list>
#include <vector>
#include "pulsatrix/module.hpp"

Include dependency graph for rms_norm_module.hpp:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes
class	pulsatrix::RMSNormModule
	y_{n,i} = gamma_i * x_{n,i} / rms(x_n), rms(x_n) = sqrt(mean_i(x_{n,i}^2) + eps), computed independently per batch row n. Batched ((N, num_features)), migrated from the original unbatched (rank-1) scope by campaign_exai_dl_library_batch_dimension_support – no mean-centering, no beta/bias term (RMSNorm's defining simplification vs. LayerNorm). More...

Namespaces
namespace	pulsatrix

Detailed Description

RMS normalization layer (Zhang & Sennrich, 2019) – this codebase's first normalization Module, establishing the Tier 1 pattern for GroupNorm/LayerNorm.

Classes

Namespaces

Detailed Description