pulsatrix 1.1.0
Loading...
Searching...
No Matches
transformer_block.hpp File Reference

Pre-LN transformer block – Phase 3's literal exit-gate deliverable, third and last composition mission (RMSNorm x2, MultiHeadAttentionModule, SwiGLUModule, plus the resolved residual-split LRP rule). More...

#include <memory>
#include <optional>
#include "pulsatrix/mlp_hook.hpp"
#include "pulsatrix/module.hpp"
#include "pulsatrix/multihead_attention_module.hpp"
#include "pulsatrix/rms_norm_module.hpp"
#include "pulsatrix/swiglu_module.hpp"
Include dependency graph for transformer_block.hpp:
This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes

struct  pulsatrix::TransformerBlockOptions
 The parts of a TransformerBlock beyond its attention (LLM-9 adds Gemma's). More...
 
class  pulsatrix::TransformerBlock
 y1 = x + MHA(RMSNorm(x)), y2 = y1 + SwiGLU(RMSNorm(y1)). Shape (N, L, d_model) -> (N, L, d_model). More...
 

Namespaces

namespace  pulsatrix
 

Detailed Description

Pre-LN transformer block – Phase 3's literal exit-gate deliverable, third and last composition mission (RMSNorm x2, MultiHeadAttentionModule, SwiGLUModule, plus the resolved residual-split LRP rule).