|
| | TransformerBlock (int64_t d_model, int64_t num_heads, int64_t d_ff, DeviceBackend *backend, bool use_rope=true, bool use_qk_norm=false) |
| | Constructs a transformer block with zero-initialized sub-module parameters.
|
| |
| | TransformerBlock (const AttentionConfig &attention, int64_t d_ff, DeviceBackend *backend, float norm_eps=1e-6f, bool mlp_bias=true) |
| | Constructs a block whose attention is configured in full (LLM-1): grouped-query attention, a causal mask, the RoPE layout and so on. See AttentionConfig.
|
| |
| | TransformerBlock (const AttentionConfig &attention, int64_t d_ff, DeviceBackend *backend, const TransformerBlockOptions &options) |
| | A block with every option (LLM-9): Gemma's GeGLU MLP, sandwich norms and 1 + w norms. With default options it is the block above.
|
| |
| Tensor | backward (const Tensor &grad_output) override |
| | Gradient w.r.t. this module's input; sub-module parameter gradients accumulate inside norm1_/mha_/norm2_/swiglu_ (reachable through parameters()).
|
| |
| OpType | op_type () const override |
| | Elementwise per this module's own op_type() note above.
|
| |
| Tensor | propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override |
| | LRP relevance propagation: two residual epsilon/z-rule splits composed with norm1_'s/mha_'s/norm2_'s/swiglu_'s own propagate_relevance().
|
| |
| std::vector< NamedParamRef > | named_parameters () override |
| | norm1_'s, mha_'s, norm2_'s, and swiglu_'s parameters, flattened.
|
| |
| std::vector< NamedModuleRef > | named_children () override |
| | This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
|
| |
| std::vector< NamedBufferRef > | named_buffers () override |
| | Every layer's named_buffers(), prefixed the same way as named_parameters().
|
| |
| void | set_training (bool training) override |
| | Cascades to every sub-module, the same way SequentialModule/MultiHeadAttentionModule do.
|
| |
| int64_t | d_model () const |
| |
| Tensor | forward_cached (const Tensor &input, KVCache &cache) |
| | The block for new positions only, with attention against cache (LLM-5); see MultiHeadAttentionModule::forward_cached. Inference only: backward() and propagate_relevance() throw until the next forward().
|
| |
| std::optional< DeviceType > | compute_device () const override |
| | Where this layer computes, so forward() rejects an input on another device (FND-8).
|
| |
|
| RMSNormModule & | norm1 () |
| |
| MultiHeadAttentionModule & | mha () |
| |
| RMSNormModule & | norm2 () |
| |
| SwiGLUModule & | swiglu () |
| |
| void | set_mlp_hook (MlpHook hook) |
| | Calls hook on the MLP's input (after norm2) and output (after any post-MLP norm) during forward(), not forward_cached() (MlpHook). An empty function removes it.
|
| |
| RMSNormModule * | post_attn_norm () |
| | The sandwich norms after attention and after the MLP; nullptr without post_norms.
|
| |
| RMSNormModule * | post_mlp_norm () |
| |
| virtual | ~Module ()=default |
| |
| Tensor | forward (const Tensor &input) |
| | Runs this module's forward computation.
|
| |
| std::pair< Tensor, NodeId > | forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd) |
| | Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
|
| |
| virtual bool | supports_lrp_rule (LRPRule rule) const |
| | Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
|
| |
| virtual std::vector< ParamRef > | parameters () |
| | This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
|
| |
| void | set_requires_grad (bool requires_grad, const std::string &prefix="") |
| | Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
|
| |
| bool | is_training () const |
| | Whether this module is currently in training mode.
|
| |
| virtual void | release_activations () |
| | Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
|
| |
y1 = x + MHA(RMSNorm(x)), y2 = y1 + SwiGLU(RMSNorm(y1)). Shape (N, L, d_model) -> (N, L, d_model).
Composition over inheritance, third exercise of this principle after MultiHeadAttentionModule and SwiGLUModule: norm1_/norm2_/mha_/swiglu_ are real sub-objects driven through their own forward()/backward()/propagate_relevance(). The only hand-written operation is the residual add, using this codebase's own two-term weighted-sum epsilon/z-rule (the same shape LSTMModule's c_t = f_t*c_{t-1}+i_t*g_t and GRUModule's h_t = (1-z_t)*h_{t-1}+z_t*n_t already use, weight fixed at 1 instead of a gate value) – AttnLRP does not itself address residual connections; this is mission_transformer_block.md's own resolution, not a new rule invented ad hoc.
- Note
- **
RMSNormModule, not LayerNormModule**, for stylistic consistency with this block's other LLaMA-family choices (SwiGLU, RoPE, optional QK-Norm) – not a technical requirement; both are AttnLRP Eq. 19 identity-pass-through and would work identically here.
-
op_type() reuses OpType::Elementwise – the residual add is a plain elementwise binary op with fixed weight 1, the same category RoPEModule and SwiGLUModule's gate multiply already reuse. Not Composite (this module owns real math, the residual split, disqualifying it by MultiHeadAttentionModule's own precedent) and not a new category (unlike attention's genuinely novel cross-position mixing, a residual add is not architecturally novel).
-
Conservation is dominated by
MultiHeadAttentionModule's own known large gap (measured ~49.6% of its own output in mission_multihead_attention.md), propagated through unchanged by the two residual splits (which conserve near-exactly, same argument as SwiGLUModule's diagonal split) and by SwiGLUModule's own near-exact contribution. Measured and decomposed stage-by-stage in tests/transformer_block_test.cpp, not assumed.
- See also
- cpp_engineering.aDNA's what/context/cpp_tdd/context_tdd_lrp_rule_pattern_taxonomy.md, "Known-Non-Conserving-by-Design Note" – this block's gap is inherited from MultiHeadAttentionModule, not a new instance of the exception.