pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::MultiHeadAttentionModule Class Reference

softmax(Q @ K^T / sqrt(head_dim)) @ V, multi-head, with optional RoPE and optional QK-Norm. Shape (N, L, d_model) -> (N, L, d_model). More...

#include <multihead_attention_module.hpp>

Inheritance diagram for pulsatrix::MultiHeadAttentionModule:
Collaboration diagram for pulsatrix::MultiHeadAttentionModule:

Public Member Functions

 MultiHeadAttentionModule (int64_t d_model, int64_t num_heads, DeviceBackend *backend, bool use_rope=true, bool use_qk_norm=false)
 Constructs a multi-head attention block with zero-initialized projections.
 
 MultiHeadAttentionModule (const AttentionConfig &config, DeviceBackend *backend)
 Constructs attention from a full AttentionConfig, with zero-initialized projections.
 
Tensor backward (const Tensor &grad_output) override
 Gradient w.r.t. this module's input; sub-module parameter gradients accumulate inside those sub-modules (reachable through parameters()).
 
OpType op_type () const override
 Attention per the charter's closed OpType set – see OpType::Attention's own note.
 
Tensor propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override
 LRP relevance propagation, composed from the sub-modules' own rules plus AttnLRP's Eq. 15 bilinear ("uniform") rule for the two batched matmuls.
 
const Tensor & key_relevance () const
 The last propagate_relevance()'s relevance at the K and V projection outputs, (N * L, num_kv_heads * head_dim), after each shared head has summed its query heads' relevance (LLM-7 checks the per-head totals against LXT). RoPE and QK-Norm have already been propagated through, so K's is at k_proj's output, where a Hugging Face hook on k_proj sees it.
 
const Tensor & value_relevance () const
 The same at the V projection output.
 
std::vector< NamedParamRef > named_parameters () override
 Every sub-module's parameters, flattened – Q/K/V/O weights and biases, plus the two QK-Norm gammas when enabled. RoPE and softmax contribute none.
 
std::vector< NamedModuleRef > named_children () override
 This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
 
std::vector< NamedBufferRef > named_buffers () override
 Every layer's named_buffers(), prefixed the same way as named_parameters().
 
void set_training (bool training) override
 Cascades to every sub-module, the same way SequentialModule does.
 
int64_t d_model () const
 
int64_t num_heads () const
 
int64_t num_kv_heads () const
 
int64_t head_dim () const
 
bool uses_rope () const
 
bool uses_qk_norm () const
 
bool is_causal () const
 
const AttentionConfig & config () const
 The configuration with its 0 defaults resolved (num_kv_heads, head_dim).
 
void set_key_padding_mask (const Tensor &key_keep)
 Masks padding keys in every following forward(), until cleared (LLM-1).
 
void clear_key_padding_mask ()
 Removes the key padding mask.
 
bool has_key_padding_mask () const
 
void set_position_offset (int64_t offset)
 Position of the input's first token, for RoPE (LLM-1). Forward rotates positions offset .. offset + L - 1. It does not change the mask: queries and keys are the same L tokens.
 
int64_t position_offset () const
 
Tensor forward_cached (const Tensor &input, KVCache &cache)
 Attention for new positions only, against everything in cache (LLM-5): projects the input, writes its keys and values into the cache after the positions already there, and attends over all of them. Its RoPE positions start at position_offset() + cache.length(), and the causal mask lets new position i see cached positions up to its own.
 
KVCache MakeKVCache (int64_t batch, int64_t max_length) const
 An empty cache sized for this layer: (batch, num_kv_heads, max_length, head_dim).
 
const Tensor & last_attention_weights () const
 Cached attention weights of the last forward, (N, num_heads, L, L) – the softmax output. Exposed because "what did each head attend to" is the single most-asked explainability question about this module. Masked entries are 0.
 
std::optional< DeviceType > compute_device () const override
 Where this layer computes, so forward() rejects an input on another device (FND-8).
 
void release_activations () override
 Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
 
Sub-module access – weight initialization from tests/loaders, and inspection.
LinearModule & q_proj ()
 
LinearModule & k_proj ()
 
LinearModule & v_proj ()
 
LinearModule & out_proj ()
 
RMSNormModule * q_norm ()
 Q's QK-Norm sub-module, or nullptr when use_qk_norm is false.
 
RMSNormModule * k_norm ()
 K's QK-Norm sub-module, or nullptr when use_qk_norm is false.
 
- Public Member Functions inherited from pulsatrix::Module
virtual ~Module ()=default
 
Tensor forward (const Tensor &input)
 Runs this module's forward computation.
 
std::pair< Tensor, NodeId > forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd)
 Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
 
virtual bool supports_lrp_rule (LRPRule rule) const
 Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
 
virtual std::vector< ParamRef > parameters ()
 This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
 
void set_requires_grad (bool requires_grad, const std::string &prefix="")
 Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
 
bool is_training () const
 Whether this module is currently in training mode.
 

Protected Member Functions

Tensor forward_impl (const Tensor &input) override
 Runs the 9-step pipeline: project -> split heads -> (QK-Norm) -> (RoPE) -> scores -> softmax -> context -> merge heads -> output projection.
 

Detailed Description

softmax(Q @ K^T / sqrt(head_dim)) @ V, multi-head, with optional RoPE and optional QK-Norm. Shape (N, L, d_model) -> (N, L, d_model).

LLM-1 additions, all configured through AttentionConfig: grouped-query attention, head_dim independent of d_model, optional projection biases, a causal mask, a key padding mask (set_key_padding_mask), the RoPE pair layout and base, and a position offset (set_position_offset).

Note
Grouped-query attention repeats K and V, as Hugging Face's repeat_kv does: K/V are projected, normalized and rotated at num_kv_heads, then copied out to num_heads before the two matmuls. backward() and propagate_relevance() sum each copy's gradient or relevance back into its shared head, so a shared head's relevance is the total over the query heads that read it.
Masked scores are set to the lowest float before softmax, not -inf, so a query with every key masked (for example a padding query under a causal mask) gets uniform weights instead of NaN – what Hugging Face does too. The gradient and the relevance of every masked score are then set to exactly zero: a masked score is a constant, not a function of Q and K.

Composition over inheritance. Every sub-operation that already exists as a shipped, tested Module in this codebase is held here as a real member object and driven through its own forward()/backward()/propagate_relevance():

  • LinearModule q_proj_/k_proj_/v_proj_/out_proj_ (d_model -> num_heads * head_dim, d_model -> num_kv_heads * head_dim twice, and num_heads * head_dim -> d_model),
  • RoPEModule q_rope_/k_rope_ (only when use_rope),
  • RMSNormModule q_norm_/k_norm_ (only when use_qk_norm, head_dim-sized gamma),
  • SoftmaxModule softmax_. None of their math is reimplemented here. What is implemented here is only what has no existing module: the two batched matmuls (Q@K^T, Attn@V), the 1/sqrt(head_dim) scale, and the head split/merge permutation – with their own forward, backward and LRP rule.
Note
Two RoPE instances and two QK-Norm instances, not one shared each. Both of those module types cache their own forward input/output for use by backward() and propagate_relevance(). A single shared instance applied to Q and then to K would leave only K's activations in the cache, so the subsequent Q backward/relevance pass would silently redistribute through K's numbers. Separate instances are the only correct choice given those modules' caching contract (and, for QK-Norm, it also matches the usual published formulation, which gives Q and K independent gammas).
QK-Norm gamma is initialized to 1.0, not to RMSNormModule's own zero default. A zero gamma would make QK-Norm annihilate Q and K entirely, so use_qk_norm=true on a freshly constructed module would produce uniform attention regardless of the input – a silently degenerate configuration rather than a neutral default. Every other parameter here keeps this codebase's zero-init convention.
**use_rope=true requires an even head_dim** – RoPEModule rotates adjacent feature pairs and rejects an odd dimension. Checked here, before the sub-module is constructed, so the error message names the real cause.

Constructor & Destructor Documentation

◆ MultiHeadAttentionModule() [1/2]

pulsatrix::MultiHeadAttentionModule::MultiHeadAttentionModule ( int64_t  d_model,
int64_t  num_heads,
DeviceBackend *  backend,
bool  use_rope = true,
bool  use_qk_norm = false 
)

Constructs a multi-head attention block with zero-initialized projections.

Parameters
d_modelModel/embedding width. Input and output are both (N, L, d_model).
num_headsNumber of attention heads; head_dim = d_model / num_heads.
backendBackend to allocate/compute through. Not owned; must outlive this module.
use_ropeApply RoPE to Q and K after the projections (and after QK-Norm).
use_qk_normApply RMSNorm over each head's head_dim features of Q and K.
Exceptions
std::invalid_argumentif d_model <= 0, num_heads <= 0, d_model % num_heads != 0, or use_rope with an odd head_dim – external boundary (constructor arguments can originate from Phase 5's Python bindings with no upstream validation), same convention as every other module.

◆ MultiHeadAttentionModule() [2/2]

pulsatrix::MultiHeadAttentionModule::MultiHeadAttentionModule ( const AttentionConfig &  config,
DeviceBackend *  backend 
)

Constructs attention from a full AttentionConfig, with zero-initialized projections.

Exceptions
std::invalid_argumentif d_model or num_heads is not positive, num_kv_heads or head_dim is negative, num_heads is not a multiple of num_kv_heads, head_dim is 0 and d_model % num_heads != 0, or use_rope with an odd head_dim.

Member Function Documentation

◆ backward()

Tensor pulsatrix::MultiHeadAttentionModule::backward ( const Tensor &  grad_output)
overridevirtual

Gradient w.r.t. this module's input; sub-module parameter gradients accumulate inside those sub-modules (reachable through parameters()).

Parameters
grad_outputGradient w.r.t. this module's output, shape (N, L, d_model) matching the cached forward shape.
Returns
Gradient w.r.t. this module's input, shape (N, L, d_model) – the sum of the three paths (through Q, through K, through V) back into the shared input.
Note
The only hand-written gradient math here is the two batched matmuls (dA = dC @ B^T, dB = A^T @ dC, per (n, h) slice) and the head split/merge inverses (pure data movement). Everything else is a real sub-module backward() call. Verified against central finite differences over the input and every sub-module parameter in tests/multihead_attention_module_test.cpp.
Exceptions
std::logic_errorif called before any forward().
std::invalid_argumentif grad_output's shape differs from the cached forward shape.
Note
Device-generic: runs on Cpu, Cuda or Hip tensors (GPU-native-kernels Mission 2).

Implements pulsatrix::Module.

◆ clear_key_padding_mask()

void pulsatrix::MultiHeadAttentionModule::clear_key_padding_mask ( )

Removes the key padding mask.

◆ compute_device()

std::optional< DeviceType > pulsatrix::MultiHeadAttentionModule::compute_device ( ) const
inlineoverridevirtual

Where this layer computes, so forward() rejects an input on another device (FND-8).

Reimplemented from pulsatrix::Module.

◆ config()

const AttentionConfig & pulsatrix::MultiHeadAttentionModule::config ( ) const
inline

The configuration with its 0 defaults resolved (num_kv_heads, head_dim).

◆ d_model()

int64_t pulsatrix::MultiHeadAttentionModule::d_model ( ) const
inline

◆ forward_cached()

Tensor pulsatrix::MultiHeadAttentionModule::forward_cached ( const Tensor &  input,
KVCache &  cache 
)

Attention for new positions only, against everything in cache (LLM-5): projects the input, writes its keys and values into the cache after the positions already there, and attends over all of them. Its RoPE positions start at position_offset() + cache.length(), and the causal mask lets new position i see cached positions up to its own.

Parameters
input(N, L_new, d_model); the N must match the cache's batch.
cacheSized for this layer (MakeKVCache); advanced by L_new.
Returns
(N, L_new, d_model): the same as the last L_new rows of forward() over the whole sequence.
Note
Inference only: it leaves no state for backward() or propagate_relevance(), which throw until the next forward(). Grouped-query heads read their shared K/V head in place, without the copies forward() makes. A key padding mask, if set, must cover every cached and new position: (N, cache.length() + L_new). A padding query with no visible key spreads uniform weights over the keys that exist so far, so its (meaningless) output can differ from a full pass's; every real position matches.
Pieces match a full pass when attention is causal. Without the causal mask a full pass lets early positions see later ones, which a cache can't.
Exceptions
std::invalid_argumentif the input's shape, the cache's dimensions or the padding mask don't fit, or the cache would overflow.

◆ forward_impl()

Tensor pulsatrix::MultiHeadAttentionModule::forward_impl ( const Tensor &  input)
overrideprotectedvirtual

Runs the 9-step pipeline: project -> split heads -> (QK-Norm) -> (RoPE) -> scores -> softmax -> context -> merge heads -> output projection.

Parameters
input(N, L, d_model), any device.
Returns
(N, L, d_model).
Exceptions
std::invalid_argumentif input is not rank-3 with a final dimension of d_model, or a key padding mask is set whose shape is not (N, L).

Implements pulsatrix::Module.

◆ has_key_padding_mask()

bool pulsatrix::MultiHeadAttentionModule::has_key_padding_mask ( ) const
inline

◆ head_dim()

int64_t pulsatrix::MultiHeadAttentionModule::head_dim ( ) const
inline

◆ is_causal()

bool pulsatrix::MultiHeadAttentionModule::is_causal ( ) const
inline

◆ k_norm()

RMSNormModule * pulsatrix::MultiHeadAttentionModule::k_norm ( )
inline

K's QK-Norm sub-module, or nullptr when use_qk_norm is false.

◆ k_proj()

LinearModule & pulsatrix::MultiHeadAttentionModule::k_proj ( )
inline

◆ key_relevance()

const Tensor & pulsatrix::MultiHeadAttentionModule::key_relevance ( ) const

The last propagate_relevance()'s relevance at the K and V projection outputs, (N * L, num_kv_heads * head_dim), after each shared head has summed its query heads' relevance (LLM-7 checks the per-head totals against LXT). RoPE and QK-Norm have already been propagated through, so K's is at k_proj's output, where a Hugging Face hook on k_proj sees it.

Exceptions
std::logic_errorbefore any propagate_relevance().

◆ last_attention_weights()

const Tensor & pulsatrix::MultiHeadAttentionModule::last_attention_weights ( ) const
inline

Cached attention weights of the last forward, (N, num_heads, L, L) – the softmax output. Exposed because "what did each head attend to" is the single most-asked explainability question about this module. Masked entries are 0.

◆ MakeKVCache()

KVCache pulsatrix::MultiHeadAttentionModule::MakeKVCache ( int64_t  batch,
int64_t  max_length 
) const

An empty cache sized for this layer: (batch, num_kv_heads, max_length, head_dim).

◆ named_buffers()

std::vector< NamedBufferRef > pulsatrix::MultiHeadAttentionModule::named_buffers ( )
overridevirtual

Every layer's named_buffers(), prefixed the same way as named_parameters().

Reimplemented from pulsatrix::Module.

◆ named_children()

std::vector< NamedModuleRef > pulsatrix::MultiHeadAttentionModule::named_children ( )
overridevirtual

This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.

Note
The containers that build language models and their layers report their children: SequentialModule, ResidualModule, the attention, MLP and transformer blocks, CausalLM and EncoderLM. A module that doesn't is a leaf to NamedModules().

Reimplemented from pulsatrix::Module.

◆ named_parameters()

std::vector< NamedParamRef > pulsatrix::MultiHeadAttentionModule::named_parameters ( )
overridevirtual

Every sub-module's parameters, flattened – Q/K/V/O weights and biases, plus the two QK-Norm gammas when enabled. RoPE and softmax contribute none.

Reimplemented from pulsatrix::Module.

◆ num_heads()

int64_t pulsatrix::MultiHeadAttentionModule::num_heads ( ) const
inline

◆ num_kv_heads()

int64_t pulsatrix::MultiHeadAttentionModule::num_kv_heads ( ) const
inline

◆ op_type()

OpType pulsatrix::MultiHeadAttentionModule::op_type ( ) const
inlineoverridevirtual

Attention per the charter's closed OpType set – see OpType::Attention's own note.

Implements pulsatrix::Module.

◆ out_proj()

LinearModule & pulsatrix::MultiHeadAttentionModule::out_proj ( )
inline

◆ position_offset()

int64_t pulsatrix::MultiHeadAttentionModule::position_offset ( ) const
inline

◆ propagate_relevance()

Tensor pulsatrix::MultiHeadAttentionModule::propagate_relevance ( const Tensor &  relevance_out,
const LRPRuleConfig &  config 
)
overridevirtual

LRP relevance propagation, composed from the sub-modules' own rules plus AttnLRP's Eq. 15 bilinear ("uniform") rule for the two batched matmuls.

For O = A @ B per (n, h) slice, Eq. 15 (Achtibat et al. 2024) splits each output's relevance evenly between the two operands – hence the factor 2 in the denominator, which this codebase's additive epsilon rule (LinearModule/RNNModule/RoPEModule) does not have: R_A[i,j] += sum_k (A[i,j]*B[j,k] / (2*O[i,k] + eps*sign(O[i,k]))) * R_O[i,k] R_B[j,k] += sum_i (A[i,j]*B[j,k] / (2*O[i,k] + eps*sign(O[i,k]))) * R_O[i,k] Applied once to context = Attn @ V and once to scores_raw = Q @ K^T (there B is K^T, so the resulting R_B is transposed back into K's layout).

Parameters
relevance_outRelevance at this module's output, (N, L, d_model).
configSupplies the epsilon stabilizer for Eq. 15 and for the projections' epsilon rule.
Returns
Relevance at this module's input, (N, L, d_model) – summed over the Q/K/V paths.
Note
This composed rule does not conserve relevance, and is deliberately excluded from tests/lrp_conservation_test.cpp's AllModuleTypeCases() for the same reason SoftmaxModule is: the softmax step in the middle of the pipeline is AttnLRP Eq. 13, a first-order DTD approximation with a known residual "hidden bias term" (see softmax_module.hpp). Eq. 15 is itself only conserving in the sense that the two operands' shares sum back to R_O – and only when the pipeline feeding it conserves. The composed block's actual measured gap is reported by tests/multihead_attention_module_test.cpp's dedicated measurement test rather than asserted away or forced into the shared tolerance.
See also
cpp_engineering.aDNA's what/context/cpp_tdd/context_tdd_lrp_rule_pattern_taxonomy.md, "Known-Non-Conserving-by-Design Note".
Note
The 1/sqrt(head_dim) scale is a positive constant factor, under which the epsilon rule is exactly the identity (x*c/(c*x) == 1), so relevance at the scaled scores equals relevance at the raw product and Eq. 15 is applied against the cached raw Q @ K^T – no separate scale step, and no scale-dependent relevance.
Exceptions
std::logic_errorif called before any forward().
std::invalid_argumentif relevance_out's shape differs from the cached forward shape.
Note
Device-generic: runs on Cpu, Cuda or Hip tensors (GPU-native-kernels Mission 3).

Implements pulsatrix::Module.

◆ q_norm()

RMSNormModule * pulsatrix::MultiHeadAttentionModule::q_norm ( )
inline

Q's QK-Norm sub-module, or nullptr when use_qk_norm is false.

◆ q_proj()

LinearModule & pulsatrix::MultiHeadAttentionModule::q_proj ( )
inline

◆ release_activations()

void pulsatrix::MultiHeadAttentionModule::release_activations ( )
overridevirtual

Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).

Note
Until the next forward(), backward() and relevance calls are invalid. The default has nothing to free.

Reimplemented from pulsatrix::Module.

◆ set_key_padding_mask()

void pulsatrix::MultiHeadAttentionModule::set_key_padding_mask ( const Tensor &  key_keep)

Masks padding keys in every following forward(), until cleared (LLM-1).

Parameters
key_keep(N, L), any device: 1 for a real token, 0 for padding – Hugging Face's attention_mask. Its N and L must match the next forward()'s input.
Note
Only keys are masked. A padding query still produces an output row (attending to the real keys); the caller ignores it, as the loss does through its token weights.
Exceptions
std::invalid_argumentif key_keep is not rank-2.

◆ set_position_offset()

void pulsatrix::MultiHeadAttentionModule::set_position_offset ( int64_t  offset)

Position of the input's first token, for RoPE (LLM-1). Forward rotates positions offset .. offset + L - 1. It does not change the mask: queries and keys are the same L tokens.

Exceptions
std::invalid_argumentif offset is negative.

◆ set_training()

void pulsatrix::MultiHeadAttentionModule::set_training ( bool  training)
overridevirtual

Cascades to every sub-module, the same way SequentialModule does.

Reimplemented from pulsatrix::Module.

◆ uses_qk_norm()

bool pulsatrix::MultiHeadAttentionModule::uses_qk_norm ( ) const
inline

◆ uses_rope()

bool pulsatrix::MultiHeadAttentionModule::uses_rope ( ) const
inline

◆ v_proj()

LinearModule & pulsatrix::MultiHeadAttentionModule::v_proj ( )
inline

◆ value_relevance()

const Tensor & pulsatrix::MultiHeadAttentionModule::value_relevance ( ) const

The same at the V projection output.

Exceptions
std::logic_errorbefore any propagate_relevance().

The documentation for this class was generated from the following file: