pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::ComponentLinear Class Reference

A LinearModule split into C rank-one subcomponents: W ≈ V U, V (in, C) reading the input and U (C, out) writing the output, so subcomponent c is V[:, c] U[c, :] and its inner activation is h_c = (x V)_c. The bias is the target's, frozen. More...

#include <parameter_decomposition.hpp>

Inheritance diagram for pulsatrix::ComponentLinear:
Collaboration diagram for pulsatrix::ComponentLinear:

Public Member Functions

 ComponentLinear (LinearModule &target, int64_t num_components, DeviceBackend *backend, int64_t gate_hidden=16, uint64_t seed=0)
 Copies target's weight and bias, frozen.
 
int64_t in_features () const
 
int64_t out_features () const
 
int64_t num_components () const
 
int64_t gate_hidden () const
 
void use_target ()
 Later forward() calls use the target weight.
 
void use_components (std::vector< float > masks, std::vector< float > delta_masks={})
 Later forward() calls use the subcomponents under masks (N, C), and the Δ weight under delta_masks (N,) when given. Sizes are checked at forward().
 
bool uses_target () const
 
const std::vector< float > & mask_grad () const
 dL/dm from the last backward() in components mode, (N, C).
 
const std::vector< float > & delta_mask_grad () const
 dL/dm_Δ from the last backward() with a Δ mask, (N,).
 
std::vector< float > gate_outputs ()
 The gates' outputs z (N, C) for the input of the last target-mode forward(); the decomposition turns them into causal importances.
 
void gate_backward (const std::vector< float > &grad_z)
 Backward from dL/dz (N, C) for the last gate_outputs(): into the gates' parameters and, through the inner activations, into V.
 
std::vector< float > component_weight ()
 The subcomponents' sum V U, (in, out) as LinearModule stores weights.
 
const std::vector< float > & target_weight () const
 The frozen target weight, (in, out).
 
LinearModule & V ()
 
LinearModule & U ()
 
Tensor backward (const Tensor &grad_output) override
 Computes the gradient w.r.t. this module's input, given the gradient w.r.t. its output. Must be called after forward() – uses state cached from that call.
 
Tensor propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override
 Through the weight in use: the target's, or the masked subcomponents (and Δ).
 
OpType op_type () const override
 This module's operation-category tag, for ComputationGraph node tagging.
 
std::vector< NamedParamRef > named_parameters () override
 This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).
 
std::optional< DeviceType > compute_device () const override
 The device this module computes on, so forward() can reject an input on another device before any kernel sees it (roadmap FND-8).
 
void release_activations () override
 Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
 
- Public Member Functions inherited from pulsatrix::Module
virtual ~Module ()=default
 
Tensor forward (const Tensor &input)
 Runs this module's forward computation.
 
std::pair< Tensor, NodeId > forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd)
 Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
 
virtual bool supports_lrp_rule (LRPRule rule) const
 Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
 
virtual std::vector< NamedBufferRef > named_buffers ()
 This module's buffers: state a checkpoint must save that no optimizer updates.
 
virtual std::vector< NamedModuleRef > named_children ()
 This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
 
virtual std::vector< ParamRef > parameters ()
 This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
 
void set_requires_grad (bool requires_grad, const std::string &prefix="")
 Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
 
virtual void set_training (bool training)
 Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
 
bool is_training () const
 Whether this module is currently in training mode.
 

Protected Member Functions

Tensor forward_impl (const Tensor &input) override
 The actual forward computation. Called by forward() after precondition checks.
 

Detailed Description

A LinearModule split into C rank-one subcomponents: W ≈ V U, V (in, C) reading the input and U (C, out) writing the output, so subcomponent c is V[:, c] U[c, :] and its inner activation is h_c = (x V)_c. The bias is the target's, frozen.

It runs in one of two modes, chosen before each forward():

  • Target: y = x W + b with the original weight, and the input is kept for causal importance (causal_importance()).
  • Components: y = ((x V) ⊙ m) U + b under a mask m (N, C). With a Δ mask (N,) (VPD), the rest of the weight, Δ = W - V U, is added back under it: y = ((x V) ⊙ (m - m_Δ)) U + m_Δ (x W) + b. Masks of 1 then give the target exactly.

backward() returns the input's gradient, adds to V's and U's, and keeps the masks' gradients (mask_grad(), delta_mask_grad()) for the decomposition to carry into the causal importances.

Causal importance (SPD): each subcomponent has its own small MLP from its inner activation to a scalar, z_c = w_out,c · GELU(w_in,c h_c + b_in,c) + b_out,c, hidden size gate_hidden. Parameters: V.weight, U.weight, gate.in_weight, gate.in_bias, gate.out_weight, gate.out_bias.

Constructor & Destructor Documentation

◆ ComponentLinear()

pulsatrix::ComponentLinear::ComponentLinear ( LinearModule &  target,
int64_t  num_components,
DeviceBackend *  backend,
int64_t  gate_hidden = 16,
uint64_t  seed = 0 
)

Copies target's weight and bias, frozen.

  • SPD's initialization (v1 init_As_and_Bs_): random unit V and U columns and rows, each U row then scaled by V[:, c]ᵀ W U[c, :]ᵀ, its overlap with the target.
  • The gate MLPs start as the reference's: input weights N(0, 2), output weights N(0, 1/gate_hidden), zero biases.
    Exceptions
    std::invalid_argumentif num_components or gate_hidden < 1.

Member Function Documentation

◆ backward()

Tensor pulsatrix::ComponentLinear::backward ( const Tensor &  grad_output)
overridevirtual

Computes the gradient w.r.t. this module's input, given the gradient w.r.t. its output. Must be called after forward() – uses state cached from that call.

Parameters
grad_outputGradient w.r.t. this module's output.
Returns
Gradient w.r.t. this module's input.
Note
Promoted to the base class in Phase 2 Mission 0 – every existing subclass (LinearModule/ReluModule/Conv2DModule) already implemented this exact signature independently; making it virtual lets graph-wiring code (Module::forward_traced) call it polymorphically through a Module* without knowing the concrete subclass, the same way propagate_relevance already works.

Implements pulsatrix::Module.

◆ component_weight()

std::vector< float > pulsatrix::ComponentLinear::component_weight ( )

The subcomponents' sum V U, (in, out) as LinearModule stores weights.

◆ compute_device()

std::optional< DeviceType > pulsatrix::ComponentLinear::compute_device ( ) const
inlineoverridevirtual

The device this module computes on, so forward() can reject an input on another device before any kernel sees it (roadmap FND-8).

Returns
std::nullopt (the default) skips the check: a container whose layers check their own inputs, or a user module written before this existed. Every in-tree layer returns its device; EmbeddingModule doesn't, because it reads its indices through their own backend and so accepts them from any device.

Reimplemented from pulsatrix::Module.

◆ delta_mask_grad()

const std::vector< float > & pulsatrix::ComponentLinear::delta_mask_grad ( ) const
inline

dL/dm_Δ from the last backward() with a Δ mask, (N,).

◆ forward_impl()

Tensor pulsatrix::ComponentLinear::forward_impl ( const Tensor &  input)
overrideprotectedvirtual

The actual forward computation. Called by forward() after precondition checks.

Implements pulsatrix::Module.

◆ gate_backward()

void pulsatrix::ComponentLinear::gate_backward ( const std::vector< float > &  grad_z)

Backward from dL/dz (N, C) for the last gate_outputs(): into the gates' parameters and, through the inner activations, into V.

◆ gate_hidden()

int64_t pulsatrix::ComponentLinear::gate_hidden ( ) const
inline

◆ gate_outputs()

std::vector< float > pulsatrix::ComponentLinear::gate_outputs ( )

The gates' outputs z (N, C) for the input of the last target-mode forward(); the decomposition turns them into causal importances.

Exceptions
std::logic_errorbefore any target-mode forward().

◆ in_features()

int64_t pulsatrix::ComponentLinear::in_features ( ) const
inline

◆ mask_grad()

const std::vector< float > & pulsatrix::ComponentLinear::mask_grad ( ) const
inline

dL/dm from the last backward() in components mode, (N, C).

◆ named_parameters()

std::vector< NamedParamRef > pulsatrix::ComponentLinear::named_parameters ( )
overridevirtual

This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).

Returns
{name, {value, grad}} entries pointing directly at this module's own members, in a fixed order. Names are unique within the module tree. Default: empty (a parameterless module like ReluModule needs no override).
Note
Override this, not parameters(): saving, loading, freezing by name and optimizer parameter groups all key on these names.

Reimplemented from pulsatrix::Module.

◆ num_components()

int64_t pulsatrix::ComponentLinear::num_components ( ) const
inline

◆ op_type()

OpType pulsatrix::ComponentLinear::op_type ( ) const
inlineoverridevirtual

This module's operation-category tag, for ComputationGraph node tagging.

Returns
This module's OpType (see op_type.hpp's closed set).
Note
Added in Phase 2 Mission 0 alongside backward() – lets graph-wiring code tag nodes generically through a Module* rather than switching on concrete subclass.

Implements pulsatrix::Module.

◆ out_features()

int64_t pulsatrix::ComponentLinear::out_features ( ) const
inline

◆ propagate_relevance()

Tensor pulsatrix::ComponentLinear::propagate_relevance ( const Tensor &  relevance_out,
const LRPRuleConfig &  config 
)
overridevirtual

Through the weight in use: the target's, or the masked subcomponents (and Δ).

Implements pulsatrix::Module.

◆ release_activations()

void pulsatrix::ComponentLinear::release_activations ( )
overridevirtual

Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).

Note
Until the next forward(), backward() and relevance calls are invalid. The default has nothing to free.

Reimplemented from pulsatrix::Module.

◆ target_weight()

const std::vector< float > & pulsatrix::ComponentLinear::target_weight ( ) const
inline

The frozen target weight, (in, out).

◆ U()

LinearModule & pulsatrix::ComponentLinear::U ( )
inline

◆ use_components()

void pulsatrix::ComponentLinear::use_components ( std::vector< float >  masks,
std::vector< float >  delta_masks = {} 
)

Later forward() calls use the subcomponents under masks (N, C), and the Δ weight under delta_masks (N,) when given. Sizes are checked at forward().

◆ use_target()

void pulsatrix::ComponentLinear::use_target ( )

Later forward() calls use the target weight.

◆ uses_target()

bool pulsatrix::ComponentLinear::uses_target ( ) const
inline

◆ V()

LinearModule & pulsatrix::ComponentLinear::V ( )
inline

The documentation for this class was generated from the following file: