pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::TiedLMHeadModule Class Reference

logits = x @ E^T, where E is a token embedding table (vocab, d_model) that the head shares rather than copies – Hugging Face's tie_word_embeddings (SmolLM2, Qwen3, Gemma 3). Shape (..., d_model) -> (..., vocab). More...

#include <tied_lm_head_module.hpp>

Inheritance diagram for pulsatrix::TiedLMHeadModule:
Collaboration diagram for pulsatrix::TiedLMHeadModule:

Public Member Functions

 TiedLMHeadModule (EmbeddingModule &embedding, DeviceBackend *backend)
 
Tensor backward (const Tensor &grad_output) override
 Input gradient grad @ E; adds grad^T @ x into the embedding's weight gradient unless the embedding is frozen.
 
Tensor propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override
 LRP through x @ E^T, by config.rule (Epsilon, Gamma, AlphaBeta or ZBox).
 
bool supports_lrp_rule (LRPRule) const override
 Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
 
OpType op_type () const override
 This module's operation-category tag, for ComputationGraph node tagging.
 
std::vector< NamedParamRef > named_parameters () override
 Empty: the shared table is the embedding's parameter, not the head's.
 
int64_t vocab_size () const
 
int64_t d_model () const
 
std::optional< DeviceType > compute_device () const override
 The device this module computes on, so forward() can reject an input on another device before any kernel sees it (roadmap FND-8).
 
void release_activations () override
 Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
 
- Public Member Functions inherited from pulsatrix::Module
virtual ~Module ()=default
 
Tensor forward (const Tensor &input)
 Runs this module's forward computation.
 
std::pair< Tensor, NodeId > forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd)
 Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
 
virtual std::vector< NamedBufferRef > named_buffers ()
 This module's buffers: state a checkpoint must save that no optimizer updates.
 
virtual std::vector< NamedModuleRef > named_children ()
 This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
 
virtual std::vector< ParamRef > parameters ()
 This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
 
void set_requires_grad (bool requires_grad, const std::string &prefix="")
 Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
 
virtual void set_training (bool training)
 Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
 
bool is_training () const
 Whether this module is currently in training mode.
 

Protected Member Functions

Tensor forward_impl (const Tensor &input) override
 

Detailed Description

logits = x @ E^T, where E is a token embedding table (vocab, d_model) that the head shares rather than copies – Hugging Face's tie_word_embeddings (SmolLM2, Qwen3, Gemma 3). Shape (..., d_model) -> (..., vocab).

Note
One parameter, two uses. The head holds no weight of its own: it reads the embedding's table and accumulates its gradient into the embedding's gradient buffer, so the table's gradient is the sum of both uses, as in PyTorch. named_parameters() is therefore empty – the optimizer, freezing and checkpoints see the table once, under the embedding's name. Freezing the embedding freezes the head too.
No bias, as in every tied model. An untied head is a bias-free LinearModule.
LRP is the affine layer's rule with the transposed table as its weight: every LRPRule LinearModule supports, with the same Zennit semantics. With no bias, the epsilon rule's two bias conventions agree, and it conserves relevance up to the stabilizer.
The embedding must outlive this module, and stay on the same backend.

Constructor & Destructor Documentation

◆ TiedLMHeadModule()

pulsatrix::TiedLMHeadModule::TiedLMHeadModule ( EmbeddingModule &  embedding,
DeviceBackend *  backend 
)
Parameters
embeddingThe table to share. Not owned; must outlive this module.
backendThe embedding's backend. Not owned; must outlive this module.

Member Function Documentation

◆ backward()

Tensor pulsatrix::TiedLMHeadModule::backward ( const Tensor &  grad_output)
overridevirtual

Input gradient grad @ E; adds grad^T @ x into the embedding's weight gradient unless the embedding is frozen.

Exceptions
std::logic_errorif called before any forward().
std::invalid_argumentif grad_output's shape differs from the cached output shape.

Implements pulsatrix::Module.

◆ compute_device()

std::optional< DeviceType > pulsatrix::TiedLMHeadModule::compute_device ( ) const
inlineoverridevirtual

The device this module computes on, so forward() can reject an input on another device before any kernel sees it (roadmap FND-8).

Returns
std::nullopt (the default) skips the check: a container whose layers check their own inputs, or a user module written before this existed. Every in-tree layer returns its device; EmbeddingModule doesn't, because it reads its indices through their own backend and so accepts them from any device.

Reimplemented from pulsatrix::Module.

◆ d_model()

int64_t pulsatrix::TiedLMHeadModule::d_model ( ) const
inline

◆ forward_impl()

Tensor pulsatrix::TiedLMHeadModule::forward_impl ( const Tensor &  input)
overrideprotectedvirtual
Parameters
input(..., d_model), rank >= 2.
Returns
(..., vocab).
Exceptions
std::invalid_argumentif input's rank is below 2 or its last dimension isn't d_model.

Implements pulsatrix::Module.

◆ named_parameters()

std::vector< NamedParamRef > pulsatrix::TiedLMHeadModule::named_parameters ( )
inlineoverridevirtual

Empty: the shared table is the embedding's parameter, not the head's.

Reimplemented from pulsatrix::Module.

◆ op_type()

OpType pulsatrix::TiedLMHeadModule::op_type ( ) const
inlineoverridevirtual

This module's operation-category tag, for ComputationGraph node tagging.

Returns
This module's OpType (see op_type.hpp's closed set).
Note
Added in Phase 2 Mission 0 alongside backward() – lets graph-wiring code tag nodes generically through a Module* rather than switching on concrete subclass.

Implements pulsatrix::Module.

◆ propagate_relevance()

Tensor pulsatrix::TiedLMHeadModule::propagate_relevance ( const Tensor &  relevance_out,
const LRPRuleConfig &  config 
)
overridevirtual

LRP through x @ E^T, by config.rule (Epsilon, Gamma, AlphaBeta or ZBox).

Exceptions
std::logic_errorif called before any forward().
std::invalid_argumentif relevance_out's shape differs from the cached output shape, or config's rule parameters are invalid.

Implements pulsatrix::Module.

◆ release_activations()

void pulsatrix::TiedLMHeadModule::release_activations ( )
overridevirtual

Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).

Note
Until the next forward(), backward() and relevance calls are invalid. The default has nothing to free.

Reimplemented from pulsatrix::Module.

◆ supports_lrp_rule()

bool pulsatrix::TiedLMHeadModule::supports_lrp_rule ( LRPRule  rule) const
inlineoverridevirtual

Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).

Note
Default: only LRPRule::Epsilon, the rule every module implements. Overridden by LinearModule / Conv2DModule (all rules), by parameter-free pass-through modules whose relevance rule does not read the config at all, and by SequentialModule (all layers).

Reimplemented from pulsatrix::Module.

◆ vocab_size()

int64_t pulsatrix::TiedLMHeadModule::vocab_size ( ) const
inline

The documentation for this class was generated from the following file: