pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::EncoderLM Class Reference

A masked language model: token embeddings (with ESM's token-dropout scaling, when the config has it), pre-LayerNorm encoder layers with rotary attention (EncoderBlock), a final LayerNorm, and the LM head. Shape: token ids (N, L) to logits (N, L, vocab). More...

#include <encoder_lm.hpp>

Inheritance diagram for pulsatrix::EncoderLM:
Collaboration diagram for pulsatrix::EncoderLM:

Public Types

using AttentionObserver = std::function< void(int64_t layer, const Tensor &attention)>
 Called with a layer's index and its attention probabilities, (N, heads, L, L) on the model's device, during forward().
 

Public Member Functions

 EncoderLM (const EncoderLMConfig &config, DeviceBackend *backend)
 
Tensor backward (const Tensor &grad_output) override
 
Tensor backward_hidden (const Tensor &grad_hidden)
 backward() from the gradient at last_hidden_state() (after the final LayerNorm), (N, L, hidden_size), for a head of your own on the representations (PLM-7). The LM head gets no gradient.
 
Tensor propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override
 Relevance per input token, (N, L).
 
std::vector< Tensor > propagate_relevance_by_layer (const Tensor &relevance_out, const LRPRuleConfig &config)
 propagate_relevance, stopping at every layer boundary.
 
std::vector< Tensor > propagate_hidden_relevance_by_layer (const Tensor &relevance_hidden, const LRPRuleConfig &config)
 propagate_relevance_by_layer from relevance at last_hidden_state() (after the final LayerNorm), (N, L, hidden_size), for a head of your own on the representations: a probe, a regression or a classifier (PLM-6).
 
OpType op_type () const override
 This module's operation-category tag, for ComputationGraph node tagging.
 
std::vector< NamedParamRef > named_parameters () override
 This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).
 
std::vector< NamedModuleRef > named_children () override
 This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
 
std::vector< NamedBufferRef > named_buffers () override
 This module's buffers: state a checkpoint must save that no optimizer updates.
 
void set_training (bool training) override
 Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
 
std::optional< DeviceType > compute_device () const override
 The device this module computes on, so forward() can reject an input on another device before any kernel sees it (roadmap FND-8).
 
void release_activations () override
 Frees every forward cache, last_hidden_state() and hidden_states() included.
 
void set_keep_activations (bool keep)
 With keep false, forward() keeps nothing for backward() or the relevance passes: each layer frees its activations once its output is computed, so a pass holds one layer's at a time. hidden_states() stays empty; last_hidden_state() is still set. For scoring and embedding, where only the outputs are needed. Defaults to true.
 
bool keep_activations () const
 
void set_attention_observer (AttentionObserver observer)
 Calls observer after each layer's forward, before set_keep_activations(false) frees its activations, so every attention map can be read with only one held at a time. Contacts (PLM-4) use it. An empty function removes the observer.
 
void set_hidden_state_hook (HiddenStateHook hook)
 Calls hook at every position during forward(): 0 is the scaled embeddings, i layer i's output (before the final LayerNorm for the last). An empty function removes it. hidden_states() records what the hook returned.
 
void set_padding_mask (const Tensor &keep)
 Masks padding for the following forward passes: keep is (N, L), 1 for a real token and 0 for padding, as Hugging Face's attention_mask.
 
void clear_padding_mask ()
 Treats every token as real again.
 
const EncoderLMConfig & config () const
 
int64_t num_layers () const
 
EncoderBlock & layer (int64_t i)
 
EmbeddingModule & embed_tokens ()
 
const std::vector< Tensor > & hidden_states () const
 The last forward pass's hidden states, each (N, L, hidden_size): the scaled embeddings, then every layer's output. The last entry is before the final LayerNorm. Hugging Face's hidden_states match these, except that its last entry is after it.
 
const Tensor & last_hidden_state () const
 The last forward pass's final-LayerNorm output, (N, L, hidden_size): the per-residue representations ("embeddings") usually taken from ESM.
 
- Public Member Functions inherited from pulsatrix::Module
virtual ~Module ()=default
 
Tensor forward (const Tensor &input)
 Runs this module's forward computation.
 
std::pair< Tensor, NodeId > forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd)
 Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
 
virtual bool supports_lrp_rule (LRPRule rule) const
 Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
 
virtual std::vector< ParamRef > parameters ()
 This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
 
void set_requires_grad (bool requires_grad, const std::string &prefix="")
 Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
 
bool is_training () const
 Whether this module is currently in training mode.
 

Protected Member Functions

Tensor forward_impl (const Tensor &input) override
 

Detailed Description

A masked language model: token embeddings (with ESM's token-dropout scaling, when the config has it), pre-LayerNorm encoder layers with rotary attention (EncoderBlock), a final LayerNorm, and the LM head. Shape: token ids (N, L) to logits (N, L, vocab).

ESM-2's head is dense, GELU, LayerNorm, then the embedding table transposed plus a bias; MDLM's is one linear layer (EncoderLMConfig::head).

Parameters are named embed_tokens.weight, layers.<i>.* (EncoderBlock's names) and norm.*, then for ESM's head lm_head.dense.*, lm_head.norm.* and lm_head.bias (its decoder shares the embedding table), and for a linear head lm_head.weight and lm_head.bias.

Note
Token dropout. ESM-2 was trained with masked tokens' embeddings zeroed. To match at inference, every embedding is scaled by (1 - 0.15 * 0.8) / (1 - masked fraction), where the fraction counts the sequence's real (unpadded) tokens. This is done even when nothing is masked, and the model's logits depend on it.
Padding. set_padding_mask() masks padding keys in every layer and zeroes padded embeddings, as Hugging Face does with attention_mask.
LRP.
  • The token-dropout scale is a constant, so relevance passes through it.
  • The head's bias takes its epsilon-rule share.
  • Everything else uses its module's own rule.

Member Typedef Documentation

◆ AttentionObserver

using pulsatrix::EncoderLM::AttentionObserver = std::function<void(int64_t layer, const Tensor& attention)>

Called with a layer's index and its attention probabilities, (N, heads, L, L) on the model's device, during forward().

Constructor & Destructor Documentation

◆ EncoderLM()

pulsatrix::EncoderLM::EncoderLM ( const EncoderLMConfig &  config,
DeviceBackend *  backend 
)
Exceptions
std::invalid_argumentfor invalid sizes.

Member Function Documentation

◆ backward()

Tensor pulsatrix::EncoderLM::backward ( const Tensor &  grad_output)
overridevirtual
Exceptions
std::logic_errorbefore any forward(); std::invalid_argument for a shape mismatch.

Implements pulsatrix::Module.

◆ backward_hidden()

Tensor pulsatrix::EncoderLM::backward_hidden ( const Tensor &  grad_hidden)

backward() from the gradient at last_hidden_state() (after the final LayerNorm), (N, L, hidden_size), for a head of your own on the representations (PLM-7). The LM head gets no gradient.

Returns
The gradient at the embedding table's output, as backward() returns.
Exceptions
std::logic_errorbefore any forward(); std::invalid_argument for a shape mismatch.

◆ clear_padding_mask()

void pulsatrix::EncoderLM::clear_padding_mask ( )

Treats every token as real again.

◆ compute_device()

std::optional< DeviceType > pulsatrix::EncoderLM::compute_device ( ) const
inlineoverridevirtual

The device this module computes on, so forward() can reject an input on another device before any kernel sees it (roadmap FND-8).

Returns
std::nullopt (the default) skips the check: a container whose layers check their own inputs, or a user module written before this existed. Every in-tree layer returns its device; EmbeddingModule doesn't, because it reads its indices through their own backend and so accepts them from any device.

Reimplemented from pulsatrix::Module.

◆ config()

const EncoderLMConfig & pulsatrix::EncoderLM::config ( ) const
inline

◆ embed_tokens()

EmbeddingModule & pulsatrix::EncoderLM::embed_tokens ( )
inline

◆ forward_impl()

Tensor pulsatrix::EncoderLM::forward_impl ( const Tensor &  input)
overrideprotectedvirtual
Exceptions
std::invalid_argumentunless the input is (N, L) ids in the vocabulary, or the padding mask doesn't match it.

Implements pulsatrix::Module.

◆ hidden_states()

const std::vector< Tensor > & pulsatrix::EncoderLM::hidden_states ( ) const
inline

The last forward pass's hidden states, each (N, L, hidden_size): the scaled embeddings, then every layer's output. The last entry is before the final LayerNorm. Hugging Face's hidden_states match these, except that its last entry is after it.

◆ keep_activations()

bool pulsatrix::EncoderLM::keep_activations ( ) const
inline

◆ last_hidden_state()

const Tensor & pulsatrix::EncoderLM::last_hidden_state ( ) const
inline

The last forward pass's final-LayerNorm output, (N, L, hidden_size): the per-residue representations ("embeddings") usually taken from ESM.

◆ layer()

EncoderBlock & pulsatrix::EncoderLM::layer ( int64_t  i)
inline

◆ named_buffers()

std::vector< NamedBufferRef > pulsatrix::EncoderLM::named_buffers ( )
overridevirtual

This module's buffers: state a checkpoint must save that no optimizer updates.

Returns
{name, tensor} entries, named like named_parameters(), in a fixed order. Default: empty. BatchNormModule reports its running statistics; containers prefix their layers' buffers the way they prefix parameters.

Reimplemented from pulsatrix::Module.

◆ named_children()

std::vector< NamedModuleRef > pulsatrix::EncoderLM::named_children ( )
overridevirtual

This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.

Note
The containers that build language models and their layers report their children: SequentialModule, ResidualModule, the attention, MLP and transformer blocks, CausalLM and EncoderLM. A module that doesn't is a leaf to NamedModules().

Reimplemented from pulsatrix::Module.

◆ named_parameters()

std::vector< NamedParamRef > pulsatrix::EncoderLM::named_parameters ( )
overridevirtual

This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).

Returns
{name, {value, grad}} entries pointing directly at this module's own members, in a fixed order. Names are unique within the module tree. Default: empty (a parameterless module like ReluModule needs no override).
Note
Override this, not parameters(): saving, loading, freezing by name and optimizer parameter groups all key on these names.

Reimplemented from pulsatrix::Module.

◆ num_layers()

int64_t pulsatrix::EncoderLM::num_layers ( ) const
inline

◆ op_type()

OpType pulsatrix::EncoderLM::op_type ( ) const
inlineoverridevirtual

This module's operation-category tag, for ComputationGraph node tagging.

Returns
This module's OpType (see op_type.hpp's closed set).
Note
Added in Phase 2 Mission 0 alongside backward() – lets graph-wiring code tag nodes generically through a Module* rather than switching on concrete subclass.

Implements pulsatrix::Module.

◆ propagate_hidden_relevance_by_layer()

std::vector< Tensor > pulsatrix::EncoderLM::propagate_hidden_relevance_by_layer ( const Tensor &  relevance_hidden,
const LRPRuleConfig &  config 
)

propagate_relevance_by_layer from relevance at last_hidden_state() (after the final LayerNorm), (N, L, hidden_size), for a head of your own on the representations: a probe, a regression or a classifier (PLM-6).

Exceptions
std::logic_errorbefore any forward(); std::invalid_argument for a shape mismatch.

◆ propagate_relevance()

Tensor pulsatrix::EncoderLM::propagate_relevance ( const Tensor &  relevance_out,
const LRPRuleConfig &  config 
)
overridevirtual

Relevance per input token, (N, L).

Implements pulsatrix::Module.

◆ propagate_relevance_by_layer()

std::vector< Tensor > pulsatrix::EncoderLM::propagate_relevance_by_layer ( const Tensor &  relevance_out,
const LRPRuleConfig &  config 
)

propagate_relevance, stopping at every layer boundary.

Returns
One tensor per boundary, each shaped (N, L, hidden_size): the relevance entering layer 0 (the scaled embeddings), then the relevance leaving each layer.

◆ release_activations()

void pulsatrix::EncoderLM::release_activations ( )
overridevirtual

Frees every forward cache, last_hidden_state() and hidden_states() included.

Reimplemented from pulsatrix::Module.

◆ set_attention_observer()

void pulsatrix::EncoderLM::set_attention_observer ( AttentionObserver  observer)
inline

Calls observer after each layer's forward, before set_keep_activations(false) frees its activations, so every attention map can be read with only one held at a time. Contacts (PLM-4) use it. An empty function removes the observer.

◆ set_hidden_state_hook()

void pulsatrix::EncoderLM::set_hidden_state_hook ( HiddenStateHook  hook)
inline

Calls hook at every position during forward(): 0 is the scaled embeddings, i layer i's output (before the final LayerNorm for the last). An empty function removes it. hidden_states() records what the hook returned.

◆ set_keep_activations()

void pulsatrix::EncoderLM::set_keep_activations ( bool  keep)
inline

With keep false, forward() keeps nothing for backward() or the relevance passes: each layer frees its activations once its output is computed, so a pass holds one layer's at a time. hidden_states() stays empty; last_hidden_state() is still set. For scoring and embedding, where only the outputs are needed. Defaults to true.

◆ set_padding_mask()

void pulsatrix::EncoderLM::set_padding_mask ( const Tensor &  keep)

Masks padding for the following forward passes: keep is (N, L), 1 for a real token and 0 for padding, as Hugging Face's attention_mask.

Exceptions
std::invalid_argumentunless every value is 0 or 1.

◆ set_training()

void pulsatrix::EncoderLM::set_training ( bool  training)
overridevirtual

Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).

Note
Virtual since Phase 6 Mission 6 (SequentialModule) – Mission 5 originally shipped this as plain non-virtual state ("no composite container exists yet to cascade through"); SequentialModule overrides this to cascade to every contained layer, and needs virtual dispatch to do so correctly even when accessed through a Module* base pointer, not just its own concrete type.
BatchNormModule keeps running statistics in training mode and normalizes with them in eval mode; DropoutModule is the identity in eval mode. Call set_training(false) before explaining a model that has either, so a sample's explanation doesn't depend on the rest of its batch.

Reimplemented from pulsatrix::Module.


The documentation for this class was generated from the following file: