pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::TransformerBlock Class Reference

y1 = x + MHA(RMSNorm(x)), y2 = y1 + SwiGLU(RMSNorm(y1)). Shape (N, L, d_model) -> (N, L, d_model). More...

#include <transformer_block.hpp>

Inheritance diagram for pulsatrix::TransformerBlock:
Collaboration diagram for pulsatrix::TransformerBlock:

Public Member Functions

 TransformerBlock (int64_t d_model, int64_t num_heads, int64_t d_ff, DeviceBackend *backend, bool use_rope=true, bool use_qk_norm=false)
 Constructs a transformer block with zero-initialized sub-module parameters.
 
 TransformerBlock (const AttentionConfig &attention, int64_t d_ff, DeviceBackend *backend, float norm_eps=1e-6f, bool mlp_bias=true)
 Constructs a block whose attention is configured in full (LLM-1): grouped-query attention, a causal mask, the RoPE layout and so on. See AttentionConfig.
 
 TransformerBlock (const AttentionConfig &attention, int64_t d_ff, DeviceBackend *backend, const TransformerBlockOptions &options)
 A block with every option (LLM-9): Gemma's GeGLU MLP, sandwich norms and 1 + w norms. With default options it is the block above.
 
Tensor backward (const Tensor &grad_output) override
 Gradient w.r.t. this module's input; sub-module parameter gradients accumulate inside norm1_/mha_/norm2_/swiglu_ (reachable through parameters()).
 
OpType op_type () const override
 Elementwise per this module's own op_type() note above.
 
Tensor propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override
 LRP relevance propagation: two residual epsilon/z-rule splits composed with norm1_'s/mha_'s/norm2_'s/swiglu_'s own propagate_relevance().
 
std::vector< NamedParamRef > named_parameters () override
 norm1_'s, mha_'s, norm2_'s, and swiglu_'s parameters, flattened.
 
std::vector< NamedModuleRef > named_children () override
 This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
 
std::vector< NamedBufferRef > named_buffers () override
 Every layer's named_buffers(), prefixed the same way as named_parameters().
 
void set_training (bool training) override
 Cascades to every sub-module, the same way SequentialModule/MultiHeadAttentionModule do.
 
int64_t d_model () const
 
Tensor forward_cached (const Tensor &input, KVCache &cache)
 The block for new positions only, with attention against cache (LLM-5); see MultiHeadAttentionModule::forward_cached. Inference only: backward() and propagate_relevance() throw until the next forward().
 
std::optional< DeviceType > compute_device () const override
 Where this layer computes, so forward() rejects an input on another device (FND-8).
 
Sub-module access – weight initialization from tests/loaders, and inspection.
RMSNormModule & norm1 ()
 
MultiHeadAttentionModule & mha ()
 
RMSNormModule & norm2 ()
 
SwiGLUModule & swiglu ()
 
void set_mlp_hook (MlpHook hook)
 Calls hook on the MLP's input (after norm2) and output (after any post-MLP norm) during forward(), not forward_cached() (MlpHook). An empty function removes it.
 
RMSNormModule * post_attn_norm ()
 The sandwich norms after attention and after the MLP; nullptr without post_norms.
 
RMSNormModule * post_mlp_norm ()
 
- Public Member Functions inherited from pulsatrix::Module
virtual ~Module ()=default
 
Tensor forward (const Tensor &input)
 Runs this module's forward computation.
 
std::pair< Tensor, NodeId > forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd)
 Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
 
virtual bool supports_lrp_rule (LRPRule rule) const
 Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
 
virtual std::vector< ParamRef > parameters ()
 This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
 
void set_requires_grad (bool requires_grad, const std::string &prefix="")
 Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
 
bool is_training () const
 Whether this module is currently in training mode.
 
virtual void release_activations ()
 Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
 

Protected Member Functions

Tensor forward_impl (const Tensor &input) override
 Runs: norm1 -> attention -> residual add -> norm2 -> SwiGLU -> residual add.
 

Detailed Description

y1 = x + MHA(RMSNorm(x)), y2 = y1 + SwiGLU(RMSNorm(y1)). Shape (N, L, d_model) -> (N, L, d_model).

Composition over inheritance, third exercise of this principle after MultiHeadAttentionModule and SwiGLUModule: norm1_/norm2_/mha_/swiglu_ are real sub-objects driven through their own forward()/backward()/propagate_relevance(). The only hand-written operation is the residual add, using this codebase's own two-term weighted-sum epsilon/z-rule (the same shape LSTMModule's c_t = f_t*c_{t-1}+i_t*g_t and GRUModule's h_t = (1-z_t)*h_{t-1}+z_t*n_t already use, weight fixed at 1 instead of a gate value) – AttnLRP does not itself address residual connections; this is mission_transformer_block.md's own resolution, not a new rule invented ad hoc.

Note
**RMSNormModule, not LayerNormModule**, for stylistic consistency with this block's other LLaMA-family choices (SwiGLU, RoPE, optional QK-Norm) – not a technical requirement; both are AttnLRP Eq. 19 identity-pass-through and would work identically here.
op_type() reuses OpType::Elementwise – the residual add is a plain elementwise binary op with fixed weight 1, the same category RoPEModule and SwiGLUModule's gate multiply already reuse. Not Composite (this module owns real math, the residual split, disqualifying it by MultiHeadAttentionModule's own precedent) and not a new category (unlike attention's genuinely novel cross-position mixing, a residual add is not architecturally novel).
Conservation is dominated by MultiHeadAttentionModule's own known large gap (measured ~49.6% of its own output in mission_multihead_attention.md), propagated through unchanged by the two residual splits (which conserve near-exactly, same argument as SwiGLUModule's diagonal split) and by SwiGLUModule's own near-exact contribution. Measured and decomposed stage-by-stage in tests/transformer_block_test.cpp, not assumed.
See also
cpp_engineering.aDNA's what/context/cpp_tdd/context_tdd_lrp_rule_pattern_taxonomy.md, "Known-Non-Conserving-by-Design Note" – this block's gap is inherited from MultiHeadAttentionModule, not a new instance of the exception.

Constructor & Destructor Documentation

◆ TransformerBlock() [1/3]

pulsatrix::TransformerBlock::TransformerBlock ( int64_t  d_model,
int64_t  num_heads,
int64_t  d_ff,
DeviceBackend *  backend,
bool  use_rope = true,
bool  use_qk_norm = false 
)

Constructs a transformer block with zero-initialized sub-module parameters.

Parameters
d_modelModel/embedding width.
num_headsNumber of attention heads; forwarded to MultiHeadAttentionModule.
d_ffSwiGLU hidden width; forwarded to SwiGLUModule.
backendBackend to allocate/compute through. Not owned; must outlive this module.
use_ropeForwarded to MultiHeadAttentionModule.
use_qk_normForwarded to MultiHeadAttentionModule.
Exceptions
std::invalid_argumentpropagated from MultiHeadAttentionModule's or SwiGLUModule's own constructors (d_model <= 0, num_heads <= 0, d_model % num_heads != 0, d_ff <= 0, odd head_dim with use_rope) – no redundant re-validation here.

◆ TransformerBlock() [2/3]

pulsatrix::TransformerBlock::TransformerBlock ( const AttentionConfig &  attention,
int64_t  d_ff,
DeviceBackend *  backend,
float  norm_eps = 1e-6f,
bool  mlp_bias = true 
)

Constructs a block whose attention is configured in full (LLM-1): grouped-query attention, a causal mask, the RoPE layout and so on. See AttentionConfig.

Parameters
attentionAttention configuration; its d_model is the block's width.
d_ffSwiGLU hidden width.
backendBackend to allocate/compute through. Not owned; must outlive this module.
norm_epsStabilizer of both RMSNorms (Hugging Face rms_norm_eps).
mlp_biasBiases on the SwiGLU projections (Hugging Face mlp_bias; Llama-family models have none).
Exceptions
std::invalid_argumentpropagated from the sub-modules' constructors.

◆ TransformerBlock() [3/3]

pulsatrix::TransformerBlock::TransformerBlock ( const AttentionConfig &  attention,
int64_t  d_ff,
DeviceBackend *  backend,
const TransformerBlockOptions &  options 
)

A block with every option (LLM-9): Gemma's GeGLU MLP, sandwich norms and 1 + w norms. With default options it is the block above.

Member Function Documentation

◆ backward()

Tensor pulsatrix::TransformerBlock::backward ( const Tensor &  grad_output)
overridevirtual

Gradient w.r.t. this module's input; sub-module parameter gradients accumulate inside norm1_/mha_/norm2_/swiglu_ (reachable through parameters()).

Parameters
grad_outputGradient w.r.t. this module's output, shape (N, L, d_model) matching the cached forward shape.
Returns
Gradient w.r.t. this module's input, same shape.
Exceptions
std::logic_errorif called before any forward().
std::invalid_argumentif grad_output's shape differs from the cached forward shape.
Note
Device-generic: runs on Cpu, Cuda or Hip tensors (GPU-native-kernels Mission 2).

Implements pulsatrix::Module.

◆ compute_device()

std::optional< DeviceType > pulsatrix::TransformerBlock::compute_device ( ) const
inlineoverridevirtual

Where this layer computes, so forward() rejects an input on another device (FND-8).

Reimplemented from pulsatrix::Module.

◆ d_model()

int64_t pulsatrix::TransformerBlock::d_model ( ) const
inline

◆ forward_cached()

Tensor pulsatrix::TransformerBlock::forward_cached ( const Tensor &  input,
KVCache &  cache 
)

The block for new positions only, with attention against cache (LLM-5); see MultiHeadAttentionModule::forward_cached. Inference only: backward() and propagate_relevance() throw until the next forward().

Parameters
input(N, L_new, d_model).

◆ forward_impl()

Tensor pulsatrix::TransformerBlock::forward_impl ( const Tensor &  input)
overrideprotectedvirtual

Runs: norm1 -> attention -> residual add -> norm2 -> SwiGLU -> residual add.

Parameters
input(N, L, d_model), any device.
Returns
(N, L, d_model).
Exceptions
std::invalid_argumentif input's rank < 2 or final dimension != d_model.

Implements pulsatrix::Module.

◆ mha()

MultiHeadAttentionModule & pulsatrix::TransformerBlock::mha ( )
inline

◆ named_buffers()

std::vector< NamedBufferRef > pulsatrix::TransformerBlock::named_buffers ( )
overridevirtual

Every layer's named_buffers(), prefixed the same way as named_parameters().

Reimplemented from pulsatrix::Module.

◆ named_children()

std::vector< NamedModuleRef > pulsatrix::TransformerBlock::named_children ( )
overridevirtual

This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.

Note
The containers that build language models and their layers report their children: SequentialModule, ResidualModule, the attention, MLP and transformer blocks, CausalLM and EncoderLM. A module that doesn't is a leaf to NamedModules().

Reimplemented from pulsatrix::Module.

◆ named_parameters()

std::vector< NamedParamRef > pulsatrix::TransformerBlock::named_parameters ( )
overridevirtual

norm1_'s, mha_'s, norm2_'s, and swiglu_'s parameters, flattened.

Reimplemented from pulsatrix::Module.

◆ norm1()

RMSNormModule & pulsatrix::TransformerBlock::norm1 ( )
inline

◆ norm2()

RMSNormModule & pulsatrix::TransformerBlock::norm2 ( )
inline

◆ op_type()

OpType pulsatrix::TransformerBlock::op_type ( ) const
inlineoverridevirtual

Elementwise per this module's own op_type() note above.

Implements pulsatrix::Module.

◆ post_attn_norm()

RMSNormModule * pulsatrix::TransformerBlock::post_attn_norm ( )
inline

The sandwich norms after attention and after the MLP; nullptr without post_norms.

◆ post_mlp_norm()

RMSNormModule * pulsatrix::TransformerBlock::post_mlp_norm ( )
inline

◆ propagate_relevance()

Tensor pulsatrix::TransformerBlock::propagate_relevance ( const Tensor &  relevance_out,
const LRPRuleConfig &  config 
)
overridevirtual

LRP relevance propagation: two residual epsilon/z-rule splits composed with norm1_'s/mha_'s/norm2_'s/swiglu_'s own propagate_relevance().

Parameters
relevance_outRelevance at this module's output, matching the cached forward shape.
configSupplies the epsilon stabilizer for the residual splits and every composed sub-module rule.
Returns
Relevance at this module's input, same shape as relevance_out.
Exceptions
std::logic_errorif called before any forward().
std::invalid_argumentif relevance_out's shape differs from the cached forward shape.
Note
Device-generic: runs on Cpu, Cuda or Hip tensors (GPU-native-kernels Mission 3).

Implements pulsatrix::Module.

◆ set_mlp_hook()

void pulsatrix::TransformerBlock::set_mlp_hook ( MlpHook  hook)
inline

Calls hook on the MLP's input (after norm2) and output (after any post-MLP norm) during forward(), not forward_cached() (MlpHook). An empty function removes it.

◆ set_training()

void pulsatrix::TransformerBlock::set_training ( bool  training)
overridevirtual

Cascades to every sub-module, the same way SequentialModule/MultiHeadAttentionModule do.

Reimplemented from pulsatrix::Module.

◆ swiglu()

SwiGLUModule & pulsatrix::TransformerBlock::swiglu ( )
inline

The documentation for this class was generated from the following file: