layer_norm_module.hpp File Reference

Layer normalization (Ba et al., 2016) – mean-centered/scaled RMSNormModule sibling. More...

#include <optional>
#include <initializer_list>
#include <vector>
#include "pulsatrix/module.hpp"

Include dependency graph for layer_norm_module.hpp:

Go to the source code of this file.

Classes
class	pulsatrix::LayerNormModule
	y_{n,i} = gamma_i * (x_{n,i} - mu_n)/std_n + beta_i, mu_n = mean_i(x_{n,i}), std_n = sqrt(var_i(x_{n,i}) + eps), computed independently per batch row n. Batched ((N, num_features)), migrated from the original unbatched (rank-1) scope by campaign_exai_dl_library_batch_dimension_support. More...

Namespaces
namespace	pulsatrix

Detailed Description

Layer normalization (Ba et al., 2016) – mean-centered/scaled RMSNormModule sibling.