transformer_block.hpp File Reference

Pre-LN transformer block – Phase 3's literal exit-gate deliverable, third and last composition mission (RMSNorm x2, MultiHeadAttentionModule, SwiGLUModule, plus the resolved residual-split LRP rule). More...

#include <optional>
#include "pulsatrix/module.hpp"
#include "pulsatrix/multihead_attention_module.hpp"
#include "pulsatrix/rms_norm_module.hpp"
#include "pulsatrix/swiglu_module.hpp"

Include dependency graph for transformer_block.hpp:

Go to the source code of this file.

Classes
class	pulsatrix::TransformerBlock
	y1 = x + MHA(RMSNorm(x)), y2 = y1 + SwiGLU(RMSNorm(y1)). Shape (N, L, d_model) -> (N, L, d_model). More...

Namespaces
namespace	pulsatrix

Detailed Description

Pre-LN transformer block – Phase 3's literal exit-gate deliverable, third and last composition mission (RMSNorm x2, MultiHeadAttentionModule, SwiGLUModule, plus the resolved residual-split LRP rule).