|
| | MultiHeadAttentionModule (int64_t d_model, int64_t num_heads, DeviceBackend *backend, bool use_rope=true, bool use_qk_norm=false) |
| | Constructs a multi-head attention block with zero-initialized projections.
|
| |
| | MultiHeadAttentionModule (const AttentionConfig &config, DeviceBackend *backend) |
| | Constructs attention from a full AttentionConfig, with zero-initialized projections.
|
| |
| Tensor | backward (const Tensor &grad_output) override |
| | Gradient w.r.t. this module's input; sub-module parameter gradients accumulate inside those sub-modules (reachable through parameters()).
|
| |
| OpType | op_type () const override |
| | Attention per the charter's closed OpType set – see OpType::Attention's own note.
|
| |
| Tensor | propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override |
| | LRP relevance propagation, composed from the sub-modules' own rules plus AttnLRP's Eq. 15 bilinear ("uniform") rule for the two batched matmuls.
|
| |
| const Tensor & | key_relevance () const |
| | The last propagate_relevance()'s relevance at the K and V projection outputs, (N * L, num_kv_heads * head_dim), after each shared head has summed its query heads' relevance (LLM-7 checks the per-head totals against LXT). RoPE and QK-Norm have already been propagated through, so K's is at k_proj's output, where a Hugging Face hook on k_proj sees it.
|
| |
| const Tensor & | value_relevance () const |
| | The same at the V projection output.
|
| |
| std::vector< NamedParamRef > | named_parameters () override |
| | Every sub-module's parameters, flattened – Q/K/V/O weights and biases, plus the two QK-Norm gammas when enabled. RoPE and softmax contribute none.
|
| |
| std::vector< NamedModuleRef > | named_children () override |
| | This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
|
| |
| std::vector< NamedBufferRef > | named_buffers () override |
| | Every layer's named_buffers(), prefixed the same way as named_parameters().
|
| |
| void | set_training (bool training) override |
| | Cascades to every sub-module, the same way SequentialModule does.
|
| |
| int64_t | d_model () const |
| |
| int64_t | num_heads () const |
| |
| int64_t | num_kv_heads () const |
| |
| int64_t | head_dim () const |
| |
| bool | uses_rope () const |
| |
| bool | uses_qk_norm () const |
| |
| bool | is_causal () const |
| |
| const AttentionConfig & | config () const |
| | The configuration with its 0 defaults resolved (num_kv_heads, head_dim).
|
| |
| void | set_key_padding_mask (const Tensor &key_keep) |
| | Masks padding keys in every following forward(), until cleared (LLM-1).
|
| |
| void | clear_key_padding_mask () |
| | Removes the key padding mask.
|
| |
| bool | has_key_padding_mask () const |
| |
| void | set_position_offset (int64_t offset) |
| | Position of the input's first token, for RoPE (LLM-1). Forward rotates positions offset .. offset + L - 1. It does not change the mask: queries and keys are the same L tokens.
|
| |
| int64_t | position_offset () const |
| |
| Tensor | forward_cached (const Tensor &input, KVCache &cache) |
| | Attention for new positions only, against everything in cache (LLM-5): projects the input, writes its keys and values into the cache after the positions already there, and attends over all of them. Its RoPE positions start at position_offset() + cache.length(), and the causal mask lets new position i see cached positions up to its own.
|
| |
| KVCache | MakeKVCache (int64_t batch, int64_t max_length) const |
| | An empty cache sized for this layer: (batch, num_kv_heads, max_length, head_dim).
|
| |
| const Tensor & | last_attention_weights () const |
| | Cached attention weights of the last forward, (N, num_heads, L, L) – the softmax output. Exposed because "what did each head attend to" is the single most-asked explainability question about this module. Masked entries are 0.
|
| |
| std::optional< DeviceType > | compute_device () const override |
| | Where this layer computes, so forward() rejects an input on another device (FND-8).
|
| |
| void | release_activations () override |
| | Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
|
| |
|
| LinearModule & | q_proj () |
| |
| LinearModule & | k_proj () |
| |
| LinearModule & | v_proj () |
| |
| LinearModule & | out_proj () |
| |
| RMSNormModule * | q_norm () |
| | Q's QK-Norm sub-module, or nullptr when use_qk_norm is false.
|
| |
| RMSNormModule * | k_norm () |
| | K's QK-Norm sub-module, or nullptr when use_qk_norm is false.
|
| |
| virtual | ~Module ()=default |
| |
| Tensor | forward (const Tensor &input) |
| | Runs this module's forward computation.
|
| |
| std::pair< Tensor, NodeId > | forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd) |
| | Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
|
| |
| virtual bool | supports_lrp_rule (LRPRule rule) const |
| | Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
|
| |
| virtual std::vector< ParamRef > | parameters () |
| | This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
|
| |
| void | set_requires_grad (bool requires_grad, const std::string &prefix="") |
| | Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
|
| |
| bool | is_training () const |
| | Whether this module is currently in training mode.
|
| |
softmax(Q @ K^T / sqrt(head_dim)) @ V, multi-head, with optional RoPE and optional QK-Norm. Shape (N, L, d_model) -> (N, L, d_model).
LLM-1 additions, all configured through AttentionConfig: grouped-query attention, head_dim independent of d_model, optional projection biases, a causal mask, a key padding mask (set_key_padding_mask), the RoPE pair layout and base, and a position offset (set_position_offset).
- Note
- Grouped-query attention repeats K and V, as Hugging Face's
repeat_kv does: K/V are projected, normalized and rotated at num_kv_heads, then copied out to num_heads before the two matmuls. backward() and propagate_relevance() sum each copy's gradient or relevance back into its shared head, so a shared head's relevance is the total over the query heads that read it.
-
Masked scores are set to the lowest float before softmax, not
-inf, so a query with every key masked (for example a padding query under a causal mask) gets uniform weights instead of NaN – what Hugging Face does too. The gradient and the relevance of every masked score are then set to exactly zero: a masked score is a constant, not a function of Q and K.
Composition over inheritance. Every sub-operation that already exists as a shipped, tested Module in this codebase is held here as a real member object and driven through its own forward()/backward()/propagate_relevance():
LinearModule q_proj_/k_proj_/v_proj_/out_proj_ (d_model -> num_heads * head_dim, d_model -> num_kv_heads * head_dim twice, and num_heads * head_dim -> d_model),
RoPEModule q_rope_/k_rope_ (only when use_rope),
RMSNormModule q_norm_/k_norm_ (only when use_qk_norm, head_dim-sized gamma),
SoftmaxModule softmax_. None of their math is reimplemented here. What is implemented here is only what has no existing module: the two batched matmuls (Q@K^T, Attn@V), the 1/sqrt(head_dim) scale, and the head split/merge permutation – with their own forward, backward and LRP rule.
- Note
- Two RoPE instances and two QK-Norm instances, not one shared each. Both of those module types cache their own forward input/output for use by
backward() and propagate_relevance(). A single shared instance applied to Q and then to K would leave only K's activations in the cache, so the subsequent Q backward/relevance pass would silently redistribute through K's numbers. Separate instances are the only correct choice given those modules' caching contract (and, for QK-Norm, it also matches the usual published formulation, which gives Q and K independent gammas).
-
QK-Norm gamma is initialized to 1.0, not to
RMSNormModule's own zero default. A zero gamma would make QK-Norm annihilate Q and K entirely, so use_qk_norm=true on a freshly constructed module would produce uniform attention regardless of the input – a silently degenerate configuration rather than a neutral default. Every other parameter here keeps this codebase's zero-init convention.
-
**
use_rope=true requires an even head_dim** – RoPEModule rotates adjacent feature pairs and rejects an odd dimension. Checked here, before the sub-module is constructed, so the error message names the real cause.
| Tensor pulsatrix::MultiHeadAttentionModule::propagate_relevance |
( |
const Tensor & |
relevance_out, |
|
|
const LRPRuleConfig & |
config |
|
) |
| |
|
overridevirtual |
LRP relevance propagation, composed from the sub-modules' own rules plus AttnLRP's Eq. 15 bilinear ("uniform") rule for the two batched matmuls.
For O = A @ B per (n, h) slice, Eq. 15 (Achtibat et al. 2024) splits each output's relevance evenly between the two operands – hence the factor 2 in the denominator, which this codebase's additive epsilon rule (LinearModule/RNNModule/RoPEModule) does not have: R_A[i,j] += sum_k (A[i,j]*B[j,k] / (2*O[i,k] + eps*sign(O[i,k]))) * R_O[i,k] R_B[j,k] += sum_i (A[i,j]*B[j,k] / (2*O[i,k] + eps*sign(O[i,k]))) * R_O[i,k] Applied once to context = Attn @ V and once to scores_raw = Q @ K^T (there B is K^T, so the resulting R_B is transposed back into K's layout).
- Parameters
-
| relevance_out | Relevance at this module's output, (N, L, d_model). |
| config | Supplies the epsilon stabilizer for Eq. 15 and for the projections' epsilon rule. |
- Returns
- Relevance at this module's input,
(N, L, d_model) – summed over the Q/K/V paths.
- Note
- This composed rule does not conserve relevance, and is deliberately excluded from tests/lrp_conservation_test.cpp's AllModuleTypeCases() for the same reason SoftmaxModule is: the softmax step in the middle of the pipeline is AttnLRP Eq. 13, a first-order DTD approximation with a known residual "hidden bias term" (see softmax_module.hpp). Eq. 15 is itself only conserving in the sense that the two operands' shares sum back to
R_O – and only when the pipeline feeding it conserves. The composed block's actual measured gap is reported by tests/multihead_attention_module_test.cpp's dedicated measurement test rather than asserted away or forced into the shared tolerance.
- See also
- cpp_engineering.aDNA's what/context/cpp_tdd/context_tdd_lrp_rule_pattern_taxonomy.md, "Known-Non-Conserving-by-Design Note".
- Note
- The
1/sqrt(head_dim) scale is a positive constant factor, under which the epsilon rule is exactly the identity (x*c/(c*x) == 1), so relevance at the scaled scores equals relevance at the raw product and Eq. 15 is applied against the cached raw Q @ K^T – no separate scale step, and no scale-dependent relevance.
- Exceptions
-
| std::logic_error | if called before any forward(). |
| std::invalid_argument | if relevance_out's shape differs from the cached forward shape. |
- Note
- Device-generic: runs on Cpu, Cuda or Hip tensors (GPU-native-kernels Mission 3).
Implements pulsatrix::Module.