pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::EncoderBlock Class Reference

One encoder layer. Shape (N, L, d_model) -> (N, L, d_model). More...

#include <encoder_block.hpp>

Inheritance diagram for pulsatrix::EncoderBlock:
Collaboration diagram for pulsatrix::EncoderBlock:

Public Member Functions

 EncoderBlock (const AttentionConfig &attention, int64_t d_ff, DeviceBackend *backend, const EncoderBlockOptions &options={})
 
Tensor backward (const Tensor &grad_output) override
 
Tensor propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override
 
OpType op_type () const override
 Owns the residual adds, as TransformerBlock does.
 
std::vector< NamedParamRef > named_parameters () override
 This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).
 
std::vector< NamedModuleRef > named_children () override
 This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
 
std::vector< NamedBufferRef > named_buffers () override
 This module's buffers: state a checkpoint must save that no optimizer updates.
 
void set_training (bool training) override
 Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
 
std::optional< DeviceType > compute_device () const override
 The device this module computes on, so forward() can reject an input on another device before any kernel sees it (roadmap FND-8).
 
int64_t d_model () const
 
const EncoderBlockOptions & options () const
 
void release_activations () override
 Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
 
Sub-modules, for loading weights and reading attention maps.
Module & norm1 ()
 
MultiHeadAttentionModule & mha ()
 
Module & norm2 ()
 
Module & mlp ()
 
void set_mlp_hook (MlpHook hook)
 Calls hook on the MLP's input and output during forward() (MlpHook). An empty function removes it.
 
- Public Member Functions inherited from pulsatrix::Module
virtual ~Module ()=default
 
Tensor forward (const Tensor &input)
 Runs this module's forward computation.
 
std::pair< Tensor, NodeId > forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd)
 Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
 
virtual bool supports_lrp_rule (LRPRule rule) const
 Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
 
virtual std::vector< ParamRef > parameters ()
 This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
 
void set_requires_grad (bool requires_grad, const std::string &prefix="")
 Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
 
bool is_training () const
 Whether this module is currently in training mode.
 

Protected Member Functions

Tensor forward_impl (const Tensor &input) override
 

Detailed Description

One encoder layer. Shape (N, L, d_model) -> (N, L, d_model).

  • Pre-norm: y1 = x + attn(norm1(x)), y2 = y1 + mlp(norm2(y1)).
  • Post-norm: y1 = norm1(x + attn(x)), y2 = norm2(y1 + mlp(y1)).

Attention is configured in full by AttentionConfig. An encoder leaves causal off; ESM-2 uses rotate-half RoPE with biases.

Parameters are named norm1.*, mha.*, norm2.* and mlp.*. A plain MLP's are mlp.fc1.* and mlp.fc2.*; a gated MLP's are mlp.gate_proj.*, mlp.up_proj.* and mlp.down_proj.*.

Note
LRP follows TransformerBlock's rules:
  • each residual add is split by the two-term epsilon rule;
  • norms and activations pass relevance through unchanged, as AttnLRP prescribes;
  • attention and linear layers use their own rules. Pre-norm with RMSNorm and a gated MLP gives the same output, gradient and relevance as TransformerBlock; the tests check this.

Constructor & Destructor Documentation

◆ EncoderBlock()

pulsatrix::EncoderBlock::EncoderBlock ( const AttentionConfig &  attention,
int64_t  d_ff,
DeviceBackend *  backend,
const EncoderBlockOptions &  options = {} 
)
Exceptions
std::invalid_argumentfrom the sub-modules' constructors, or if d_ff isn't positive.

Member Function Documentation

◆ backward()

Tensor pulsatrix::EncoderBlock::backward ( const Tensor &  grad_output)
overridevirtual
Exceptions
std::logic_errorbefore any forward(); std::invalid_argument for a shape mismatch.

Implements pulsatrix::Module.

◆ compute_device()

std::optional< DeviceType > pulsatrix::EncoderBlock::compute_device ( ) const
inlineoverridevirtual

The device this module computes on, so forward() can reject an input on another device before any kernel sees it (roadmap FND-8).

Returns
std::nullopt (the default) skips the check: a container whose layers check their own inputs, or a user module written before this existed. Every in-tree layer returns its device; EmbeddingModule doesn't, because it reads its indices through their own backend and so accepts them from any device.

Reimplemented from pulsatrix::Module.

◆ d_model()

int64_t pulsatrix::EncoderBlock::d_model ( ) const
inline

◆ forward_impl()

Tensor pulsatrix::EncoderBlock::forward_impl ( const Tensor &  input)
overrideprotectedvirtual
Exceptions
std::invalid_argumentunless the input is rank 3 with last dimension d_model.

Implements pulsatrix::Module.

◆ mha()

MultiHeadAttentionModule & pulsatrix::EncoderBlock::mha ( )
inline

◆ mlp()

Module & pulsatrix::EncoderBlock::mlp ( )
inline

◆ named_buffers()

std::vector< NamedBufferRef > pulsatrix::EncoderBlock::named_buffers ( )
overridevirtual

This module's buffers: state a checkpoint must save that no optimizer updates.

Returns
{name, tensor} entries, named like named_parameters(), in a fixed order. Default: empty. BatchNormModule reports its running statistics; containers prefix their layers' buffers the way they prefix parameters.

Reimplemented from pulsatrix::Module.

◆ named_children()

std::vector< NamedModuleRef > pulsatrix::EncoderBlock::named_children ( )
overridevirtual

This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.

Note
The containers that build language models and their layers report their children: SequentialModule, ResidualModule, the attention, MLP and transformer blocks, CausalLM and EncoderLM. A module that doesn't is a leaf to NamedModules().

Reimplemented from pulsatrix::Module.

◆ named_parameters()

std::vector< NamedParamRef > pulsatrix::EncoderBlock::named_parameters ( )
overridevirtual

This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).

Returns
{name, {value, grad}} entries pointing directly at this module's own members, in a fixed order. Names are unique within the module tree. Default: empty (a parameterless module like ReluModule needs no override).
Note
Override this, not parameters(): saving, loading, freezing by name and optimizer parameter groups all key on these names.

Reimplemented from pulsatrix::Module.

◆ norm1()

Module & pulsatrix::EncoderBlock::norm1 ( )
inline

◆ norm2()

Module & pulsatrix::EncoderBlock::norm2 ( )
inline

◆ op_type()

OpType pulsatrix::EncoderBlock::op_type ( ) const
inlineoverridevirtual

Owns the residual adds, as TransformerBlock does.

Implements pulsatrix::Module.

◆ options()

const EncoderBlockOptions & pulsatrix::EncoderBlock::options ( ) const
inline

◆ propagate_relevance()

Tensor pulsatrix::EncoderBlock::propagate_relevance ( const Tensor &  relevance_out,
const LRPRuleConfig &  config 
)
overridevirtual
Exceptions
std::logic_errorbefore any forward(); std::invalid_argument for a shape mismatch.

Implements pulsatrix::Module.

◆ release_activations()

void pulsatrix::EncoderBlock::release_activations ( )
overridevirtual

Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).

Note
Until the next forward(), backward() and relevance calls are invalid. The default has nothing to free.

Reimplemented from pulsatrix::Module.

◆ set_mlp_hook()

void pulsatrix::EncoderBlock::set_mlp_hook ( MlpHook  hook)
inline

Calls hook on the MLP's input and output during forward() (MlpHook). An empty function removes it.

◆ set_training()

void pulsatrix::EncoderBlock::set_training ( bool  training)
overridevirtual

Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).

Note
Virtual since Phase 6 Mission 6 (SequentialModule) – Mission 5 originally shipped this as plain non-virtual state ("no composite container exists yet to cascade through"); SequentialModule overrides this to cascade to every contained layer, and needs virtual dispatch to do so correctly even when accessed through a Module* base pointer, not just its own concrete type.
BatchNormModule keeps running statistics in training mode and normalizes with them in eval mode; DropoutModule is the identity in eval mode. Call set_training(false) before explaining a model that has either, so a sample's explanation doesn't depend on the rest of its batch.

Reimplemented from pulsatrix::Module.


The documentation for this class was generated from the following file: