A masked language model: token embeddings (with ESM's token-dropout scaling, when the config has it), pre-LayerNorm encoder layers with rotary attention (EncoderBlock), a final LayerNorm, and the LM head. Shape: token ids (N, L) to logits (N, L, vocab).
More...
|
| | EncoderLM (const EncoderLMConfig &config, DeviceBackend *backend) |
| |
| Tensor | backward (const Tensor &grad_output) override |
| |
| Tensor | backward_hidden (const Tensor &grad_hidden) |
| | backward() from the gradient at last_hidden_state() (after the final LayerNorm), (N, L, hidden_size), for a head of your own on the representations (PLM-7). The LM head gets no gradient.
|
| |
| Tensor | propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override |
| | Relevance per input token, (N, L).
|
| |
| std::vector< Tensor > | propagate_relevance_by_layer (const Tensor &relevance_out, const LRPRuleConfig &config) |
| | propagate_relevance, stopping at every layer boundary.
|
| |
| std::vector< Tensor > | propagate_hidden_relevance_by_layer (const Tensor &relevance_hidden, const LRPRuleConfig &config) |
| | propagate_relevance_by_layer from relevance at last_hidden_state() (after the final LayerNorm), (N, L, hidden_size), for a head of your own on the representations: a probe, a regression or a classifier (PLM-6).
|
| |
| OpType | op_type () const override |
| | This module's operation-category tag, for ComputationGraph node tagging.
|
| |
| std::vector< NamedParamRef > | named_parameters () override |
| | This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).
|
| |
| std::vector< NamedModuleRef > | named_children () override |
| | This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
|
| |
| std::vector< NamedBufferRef > | named_buffers () override |
| | This module's buffers: state a checkpoint must save that no optimizer updates.
|
| |
| void | set_training (bool training) override |
| | Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
|
| |
| std::optional< DeviceType > | compute_device () const override |
| | The device this module computes on, so forward() can reject an input on another device before any kernel sees it (roadmap FND-8).
|
| |
| void | release_activations () override |
| | Frees every forward cache, last_hidden_state() and hidden_states() included.
|
| |
| void | set_keep_activations (bool keep) |
| | With keep false, forward() keeps nothing for backward() or the relevance passes: each layer frees its activations once its output is computed, so a pass holds one layer's at a time. hidden_states() stays empty; last_hidden_state() is still set. For scoring and embedding, where only the outputs are needed. Defaults to true.
|
| |
| bool | keep_activations () const |
| |
| void | set_attention_observer (AttentionObserver observer) |
| | Calls observer after each layer's forward, before set_keep_activations(false) frees its activations, so every attention map can be read with only one held at a time. Contacts (PLM-4) use it. An empty function removes the observer.
|
| |
| void | set_hidden_state_hook (HiddenStateHook hook) |
| | Calls hook at every position during forward(): 0 is the scaled embeddings, i layer i's output (before the final LayerNorm for the last). An empty function removes it. hidden_states() records what the hook returned.
|
| |
| void | set_padding_mask (const Tensor &keep) |
| | Masks padding for the following forward passes: keep is (N, L), 1 for a real token and 0 for padding, as Hugging Face's attention_mask.
|
| |
| void | clear_padding_mask () |
| | Treats every token as real again.
|
| |
| const EncoderLMConfig & | config () const |
| |
| int64_t | num_layers () const |
| |
| EncoderBlock & | layer (int64_t i) |
| |
| EmbeddingModule & | embed_tokens () |
| |
| const std::vector< Tensor > & | hidden_states () const |
| | The last forward pass's hidden states, each (N, L, hidden_size): the scaled embeddings, then every layer's output. The last entry is before the final LayerNorm. Hugging Face's hidden_states match these, except that its last entry is after it.
|
| |
| const Tensor & | last_hidden_state () const |
| | The last forward pass's final-LayerNorm output, (N, L, hidden_size): the per-residue representations ("embeddings") usually taken from ESM.
|
| |
| virtual | ~Module ()=default |
| |
| Tensor | forward (const Tensor &input) |
| | Runs this module's forward computation.
|
| |
| std::pair< Tensor, NodeId > | forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd) |
| | Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
|
| |
| virtual bool | supports_lrp_rule (LRPRule rule) const |
| | Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
|
| |
| virtual std::vector< ParamRef > | parameters () |
| | This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
|
| |
| void | set_requires_grad (bool requires_grad, const std::string &prefix="") |
| | Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
|
| |
| bool | is_training () const |
| | Whether this module is currently in training mode.
|
| |
A masked language model: token embeddings (with ESM's token-dropout scaling, when the config has it), pre-LayerNorm encoder layers with rotary attention (EncoderBlock), a final LayerNorm, and the LM head. Shape: token ids (N, L) to logits (N, L, vocab).
ESM-2's head is dense, GELU, LayerNorm, then the embedding table transposed plus a bias; MDLM's is one linear layer (EncoderLMConfig::head).
Parameters are named embed_tokens.weight, layers.<i>.* (EncoderBlock's names) and norm.*, then for ESM's head lm_head.dense.*, lm_head.norm.* and lm_head.bias (its decoder shares the embedding table), and for a linear head lm_head.weight and lm_head.bias.
- Note
- Token dropout. ESM-2 was trained with masked tokens' embeddings zeroed. To match at inference, every embedding is scaled by
(1 - 0.15 * 0.8) / (1 - masked fraction), where the fraction counts the sequence's real (unpadded) tokens. This is done even when nothing is masked, and the model's logits depend on it.
-
Padding. set_padding_mask() masks padding keys in every layer and zeroes padded embeddings, as Hugging Face does with
attention_mask.
-
LRP.
- The token-dropout scale is a constant, so relevance passes through it.
- The head's bias takes its epsilon-rule share.
- Everything else uses its module's own rule.