y_{n,i} = gamma_i * (x_{n,i} - mu_n)/std_n + beta_i, mu_n = mean_i(x_{n,i}), std_n = sqrt(var_i(x_{n,i}) + eps), computed independently per batch row n. Batched ((N, num_features)), migrated from the original unbatched (rank-1) scope by campaign_exai_dl_library_batch_dimension_support. More...

#include <layer_norm_module.hpp>

Inheritance diagram for pulsatrix::LayerNormModule:

[legend]

Collaboration diagram for pulsatrix::LayerNormModule:

[legend]

Public Member Functions
	LayerNormModule (int64_t num_features, DeviceBackend *backend, DeviceType device, float eps=1e-6f)
	Constructs a LayerNorm layer with zero-initialized gamma and beta.

	LayerNormModule (int64_t num_features, DeviceBackend *backend)
	On backend's own device (backend->device()), default eps. Previously the device defaulted to Cpu regardless of backend (GPU-native-kernels Mission 0).

Tensor	backward (const Tensor &grad_output) override
	Computes the gradient w.r.t. this module's input, and accumulates gamma's/ beta's gradients internally.

OpType	op_type () const override
	Normalization per charter's closed OpType set.

void	set_gamma (std::initializer_list< float > values)
	Overwrites the gamma buffer – test/initialization use only.

void	set_beta (std::initializer_list< float > values)
	Overwrites the beta buffer – test/initialization use only.

void	set_gamma (const std::vector< float > &values)
	Vector overload for runtime-sized sources – see Tensor's own vector ctor.

void	set_beta (const std::vector< float > &values)
	Vector overload for runtime-sized sources – see Tensor's own vector ctor.

const Tensor &	gamma () const

const Tensor &	beta () const

const Tensor &	gamma_grad () const

const Tensor &	beta_grad () const

Tensor	propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override
	Identity-rule LRP relevance propagation (AttnLRP, Achtibat et al. 2024).

std::vector< NamedParamRef >	named_parameters () override
	This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).

std::optional< DeviceType >	compute_device () const override
	Where this layer computes, so forward() rejects an input on another device (FND-8).

Public Member Functions inherited from pulsatrix::Module
virtual	~Module ()=default

Tensor	forward (const Tensor &input)
	Runs this module's forward computation.

std::pair< Tensor, NodeId >	forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd)
	Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.

virtual bool	supports_lrp_rule (LRPRule rule) const
	Whether propagate_relevance() implements `rule` (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).

virtual std::vector< ParamRef >	parameters ()
	This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.

void	set_requires_grad (bool requires_grad, const std::string &prefix="")
	Freezes (`false`) or unfreezes (`true`) parameters by name (roadmap FND-2).

virtual void	set_training (bool training)
	Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).

bool	is_training () const
	Whether this module is currently in training mode.

Protected Member Functions
Tensor	forward_impl (const Tensor &input) override
	The actual forward computation. Called by forward() after precondition checks.

Detailed Description

y_{n,i} = gamma_i * (x_{n,i} - mu_n)/std_n + beta_i, mu_n = mean_i(x_{n,i}), std_n = sqrt(var_i(x_{n,i}) + eps), computed independently per batch row n. Batched ((N, num_features)), migrated from the original unbatched (rank-1) scope by campaign_exai_dl_library_batch_dimension_support.

Note: propagate_relevance is an identity pass-through, cited to AttnLRP's normalization-layer treatment (Achtibat et al. 2024) – identical rationale to RMSNormModule's own identical rule; see that class's Doxygen for the full citation chain (Ali et al. 2022's detach-std treatment, simplified by AttnLRP to identity). backward() is the real, undetached training gradient – the standard LayerNorm backward formula, independent of the LRP treatment.

Constructor & Destructor Documentation

◆ LayerNormModule() [1/2]

pulsatrix::LayerNormModule::LayerNormModule	(	int64_t	num_features,
		DeviceBackend *	backend,
		DeviceType	device,
		float	eps = `1e-6f`
	)

Constructs a LayerNorm layer with zero-initialized gamma and beta.

Parameters

num_features	Number of elements this layer normalizes over.
backend	Backend to allocate/compute through. Not owned; must outlive this module.
device	Which device every internal Tensor member is tagged as. Defaults to Cpu.
eps	Stabilizer added inside the sqrt. Defaults to 1e-6.

Exceptions

std::invalid_argument if num_features <= 0 – external boundary, mirrors RMSNormModule's identical constructor guard.

◆ LayerNormModule() [2/2]

pulsatrix::LayerNormModule::LayerNormModule	(	int64_t	num_features,
		DeviceBackend *	backend
	)

On backend's own device (backend->device()), default eps. Previously the device defaulted to Cpu regardless of backend (GPU-native-kernels Mission 0).

Member Function Documentation

◆ backward()

Tensor pulsatrix::LayerNormModule::backward ( const Tensor & grad_output )

overridevirtual

Computes the gradient w.r.t. this module's input, and accumulates gamma's/ beta's gradients internally.

Parameters

grad_output Gradient w.r.t. this module's output. Must match num_features.

Returns: Gradient w.r.t. this module's input.

Exceptions

std::logic_error if forward() has never been called.

Note: Device-generic: runs on Cpu, Cuda or Hip tensors (GPU-native-kernels Mission 2).

Implements pulsatrix::Module.

◆ beta()

const Tensor & pulsatrix::LayerNormModule::beta ( ) const

inline

◆ beta_grad()

const Tensor & pulsatrix::LayerNormModule::beta_grad ( ) const

inline

◆ compute_device()

std::optional< DeviceType > pulsatrix::LayerNormModule::compute_device ( ) const

inlineoverridevirtual

Where this layer computes, so forward() rejects an input on another device (FND-8).

Reimplemented from pulsatrix::Module.

◆ forward_impl()

Tensor pulsatrix::LayerNormModule::forward_impl ( const Tensor & input )

overrideprotectedvirtual

The actual forward computation. Called by forward() after precondition checks.

Implements pulsatrix::Module.

◆ gamma()

const Tensor & pulsatrix::LayerNormModule::gamma ( ) const

inline

◆ gamma_grad()

const Tensor & pulsatrix::LayerNormModule::gamma_grad ( ) const

inline

◆ named_parameters()

std::vector< NamedParamRef > pulsatrix::LayerNormModule::named_parameters ( )

inlineoverridevirtual

This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).

Returns: {name, {value, grad}} entries pointing directly at this module's own members, in a fixed order. Names are unique within the module tree. Default: empty (a parameterless module like ReluModule needs no override).

Note: Override this, not parameters(): saving, loading, freezing by name and optimizer parameter groups all key on these names.

Reimplemented from pulsatrix::Module.

◆ op_type()

OpType pulsatrix::LayerNormModule::op_type ( ) const

inlineoverridevirtual

Normalization per charter's closed OpType set.

Implements pulsatrix::Module.

◆ propagate_relevance()

Tensor pulsatrix::LayerNormModule::propagate_relevance	(	const Tensor &	relevance_out,
		const LRPRuleConfig &	config
	)

overridevirtual

Identity-rule LRP relevance propagation (AttnLRP, Achtibat et al. 2024).

Parameters

relevance_out	Relevance at this module's output. Must match num_features.
config	Unused – the identity rule has no tunable parameter.

Returns: relevance_out, unchanged – conservation holds trivially by construction.

Exceptions

std::logic_error	if forward() has never been called.
std::invalid_argument	if relevance_out's element count doesn't match num_features.

Implements pulsatrix::Module.

◆ set_beta() [1/2]

void pulsatrix::LayerNormModule::set_beta ( const std::vector< float > & values )

Vector overload for runtime-sized sources – see Tensor's own vector ctor.

◆ set_beta() [2/2]

void pulsatrix::LayerNormModule::set_beta ( std::initializer_list< float > values )

Overwrites the beta buffer – test/initialization use only.

◆ set_gamma() [1/2]

void pulsatrix::LayerNormModule::set_gamma ( const std::vector< float > & values )

Vector overload for runtime-sized sources – see Tensor's own vector ctor.

◆ set_gamma() [2/2]

void pulsatrix::LayerNormModule::set_gamma ( std::initializer_list< float > values )

Overwrites the gamma buffer – test/initialization use only.

The documentation for this class was generated from the following file:

include/pulsatrix/layer_norm_module.hpp

Public Member Functions

Protected Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ LayerNormModule() [1/2]

◆ LayerNormModule() [2/2]

Member Function Documentation

◆ backward()

◆ beta()

◆ beta_grad()

◆ compute_device()

◆ forward_impl()

◆ gamma()

◆ gamma_grad()

◆ named_parameters()

◆ op_type()

◆ propagate_relevance()

◆ set_beta() [1/2]

◆ set_beta() [2/2]

◆ set_gamma() [1/2]

◆ set_gamma() [2/2]