pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::LinearModule Class Reference

y = x @ W + b, batched (x is (N, in_features), y is (N, out_features)) – migrated from the original unbatched (rank-1) scope by campaign_exai_dl_library_batch_dimension_support (breaking migration to always-batched; a single example is N=1, not a structurally different case). More...

#include <linear_module.hpp>

Inheritance diagram for pulsatrix::LinearModule:
Collaboration diagram for pulsatrix::LinearModule:

Public Member Functions

 LinearModule (int64_t in_features, int64_t out_features, DeviceBackend *backend, DeviceType device, bool use_bias=true)
 Constructs a linear layer with zero-initialized weight/bias.
 
 LinearModule (int64_t in_features, int64_t out_features, DeviceBackend *backend, bool use_bias=true)
 As above, on backend's own device (backend->device()).
 
 ~LinearModule () override
 
 LinearModule (LinearModule &&) noexcept
 
LinearModule & operator= (LinearModule &&) noexcept
 
void attach_lora (const LoRAConfig &config)
 Adds a LoRA adapter (TRN-7, lora.hpp): from now on the layer computes x W + b + (alpha / r) x A B. A is initialized as PEFT does and B is zero, so the output doesn't change until B trains. The base weight isn't frozen here; ApplyLora() freezes it.
 
void detach_lora ()
 Removes the adapter. A merged one stays in the weight. No-op without one.
 
LoRAAdapter * lora ()
 The adapter, or nullptr.
 
const LoRAAdapter * lora () const
 
void merge_lora ()
 Adds the update into the weight, W += (alpha / r) A B, so forward() runs at the base layer's cost. While merged, A and B get no gradients.
 
void unmerge_lora ()
 Subtracts the update again. W comes back up to float32 rounding, as with PEFT.
 
void set_lora_enabled (bool enabled)
 false runs the layer as its base layer, x W + b; true adds the update again.
 
void attach_ia3 (bool feedforward)
 Adds an IA³ rescaling vector l, all ones (TRN-10, ia3.hpp): the layer then computes (x W + b) ⊙ l, or (x ⊙ l) W + b when feedforward. l joins named_parameters() as ia3_l; backward() trains it; propagate_relevance() runs on the rescaled layer, W diag(l) with bias b ⊙ l, or diag(l) W.
 
void detach_ia3 ()
 Removes the vector. A merged one stays in the weights. No-op without one.
 
IA3Adapter * ia3 ()
 
const IA3Adapter * ia3 () const
 
void merge_ia3 ()
 Folds l into the weights (and, for output scaling, the bias), as PEFT's merge() does.
 
void unmerge_ia3 ()
 Divides it back out by l + 1e-8, as PEFT's unmerge() does.
 
void set_ia3_enabled (bool enabled)
 false runs the base layer.
 
Tensor backward (const Tensor &grad_output) override
 Computes the gradient w.r.t. this module's input, and accumulates the weight/bias gradients internally (summed across the batch).
 
OpType op_type () const override
 Linear per charter's closed OpType set.
 
void set_weight (std::initializer_list< float > values)
 Overwrites the weight buffer – test/initialization use only.
 
void set_bias (std::initializer_list< float > values)
 Overwrites the bias buffer – test/initialization use only.
 
void set_weight (const std::vector< float > &values)
 Vector overload for runtime-sized sources – see Tensor's own vector ctor.
 
void set_bias (const std::vector< float > &values)
 Vector overload for runtime-sized sources – see Tensor's own vector ctor.
 
int64_t in_features () const
 
int64_t out_features () const
 
bool uses_bias () const
 False when built with use_bias = false: bias() is then all zeros.
 
const Tensor & weight () const
 
const Tensor & bias () const
 
const Tensor & weight_grad () const
 
const Tensor & bias_grad () const
 
Tensor propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override
 Epsilon-rule LRP relevance propagation (Bach et al. 2015), applied independently per example in the batch.
 
Tensor forward_with_relu (const Tensor &input, ReluModule &relu)
 relu.forward(forward(input)) with the bias add and the ReLU in one kernel (HIP-6). Both layers end with the cached state the two calls leave, so backward() and LRP are unchanged, and the values are identical. SequentialModule uses it for every LinearModule followed by a ReluModule.
 
bool supports_lrp_rule (LRPRule) const override
 Implements every LRPRule.
 
std::vector< NamedParamRef > named_parameters () override
 weight, bias (unless built without one), then a LoRA adapter's lora_A.weight and lora_B.weight, or an IA³ vector's ia3_l.
 
std::optional< DeviceType > compute_device () const override
 Where this layer computes, so forward() rejects an input on another device (FND-8).
 
void release_activations () override
 Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
 
- Public Member Functions inherited from pulsatrix::Module
virtual ~Module ()=default
 
Tensor forward (const Tensor &input)
 Runs this module's forward computation.
 
std::pair< Tensor, NodeId > forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd)
 Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
 
virtual std::vector< NamedBufferRef > named_buffers ()
 This module's buffers: state a checkpoint must save that no optimizer updates.
 
virtual std::vector< NamedModuleRef > named_children ()
 This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
 
virtual std::vector< ParamRef > parameters ()
 This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
 
void set_requires_grad (bool requires_grad, const std::string &prefix="")
 Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
 
virtual void set_training (bool training)
 Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
 
bool is_training () const
 Whether this module is currently in training mode.
 

Protected Member Functions

Tensor forward_impl (const Tensor &input) override
 The actual forward computation. Called by forward() after precondition checks.
 

Detailed Description

y = x @ W + b, batched (x is (N, in_features), y is (N, out_features)) – migrated from the original unbatched (rank-1) scope by campaign_exai_dl_library_batch_dimension_support (breaking migration to always-batched; a single example is N=1, not a structurally different case).

Note
Weight layout is (in_features, out_features), not the more common (out_features, in_features) PyTorch convention – chosen specifically so forward (x @ W) and the weight-gradient step (X^T @ grad_Y) both use DeviceBackend::gemm directly with no transpose of the weight operand. The input-gradient step (grad_Y @ W^T) and the weight-gradient step (X^T @ grad_Y, the batched sum of outer products) read their transposed operand in place through DeviceBackend::gemm_ex – no transposed copy is built.
Weights/biases are owned here as member Tensors, not ComputationGraph nodes. Parameter gradients accumulate via Tensor::accumulate() across backward() calls until something (the optimizer) resets them. Batch-dimension gradient reduction (summing weight/bias gradient contributions across the N examples in a batch) needs no new Tensor primitive – see campaign_exai_dl_library_batch_dimension_support's mission_tensor_shape_foundation.md: weight_grad's reduction falls out of gemm's own k-dimension summation (accumulated in place, beta = 1); bias_grad's is DeviceBackend::column_sums, likewise accumulated in place. Both are device-resident (GPU-native-kernels Mission 1).

Constructor & Destructor Documentation

◆ LinearModule() [1/3]

pulsatrix::LinearModule::LinearModule ( int64_t  in_features,
int64_t  out_features,
DeviceBackend *  backend,
DeviceType  device,
bool  use_bias = true 
)

Constructs a linear layer with zero-initialized weight/bias.

Parameters
in_featuresInput dimension.
out_featuresOutput dimension.
backendBackend to allocate/compute through. Not owned; must outlive this module.
deviceWhich device every internal Tensor member (weight, bias, gradients, forward-pass caches) is tagged as. Must match whatever device backend actually allocates on, or Tensor's own device-based dispatch (e.g. CopyDirection selection) will be wrong. See campaign_exai_dl_library_phase1_5_cuda_backend.md's Mission 3.
use_biasfalse builds y = x @ W with no bias parameter (LLM-1: Llama-family projections have none). The bias buffer still exists, all zeros, so every LRP rule reads it unchanged, but it is not in named_parameters(), never receives a gradient, and set_bias() throws.

◆ LinearModule() [2/3]

pulsatrix::LinearModule::LinearModule ( int64_t  in_features,
int64_t  out_features,
DeviceBackend *  backend,
bool  use_bias = true 
)

As above, on backend's own device (backend->device()).

Note
Previously the device defaulted to Cpu, so a LinearModule built on a GPU backend without an explicit tag – e.g. SwiGLUModule's three projections – held Cpu-tagged weights in device memory (GPU-native-kernels Mission 0).

◆ ~LinearModule()

pulsatrix::LinearModule::~LinearModule ( )
override

◆ LinearModule() [3/3]

pulsatrix::LinearModule::LinearModule ( LinearModule &&  )
noexcept

Member Function Documentation

◆ attach_ia3()

void pulsatrix::LinearModule::attach_ia3 ( bool  feedforward)

Adds an IA³ rescaling vector l, all ones (TRN-10, ia3.hpp): the layer then computes (x W + b) ⊙ l, or (x ⊙ l) W + b when feedforward. l joins named_parameters() as ia3_l; backward() trains it; propagate_relevance() runs on the rescaled layer, W diag(l) with bias b ⊙ l, or diag(l) W.

Exceptions
std::logic_errorif the layer already has an IA³ vector or a LoRA adapter.

◆ attach_lora()

void pulsatrix::LinearModule::attach_lora ( const LoRAConfig &  config)

Adds a LoRA adapter (TRN-7, lora.hpp): from now on the layer computes x W + b + (alpha / r) x A B. A is initialized as PEFT does and B is zero, so the output doesn't change until B trains. The base weight isn't frozen here; ApplyLora() freezes it.

Note
The adapter's A and B join named_parameters() as lora_A.weight and lora_B.weight. backward() accumulates their gradients, and propagate_relevance() runs on the adapted weight W + (alpha / r) A B.
Exceptions
std::logic_errorif the layer already has one; std::invalid_argument for invalid settings (LoRAAdapter's constructor).

◆ backward()

Tensor pulsatrix::LinearModule::backward ( const Tensor &  grad_output)
overridevirtual

Computes the gradient w.r.t. this module's input, and accumulates the weight/bias gradients internally (summed across the batch).

Parameters
grad_outputGradient w.r.t. this module's output. Must be (N, out_features), with N matching the most recent forward() call's batch size.
Returns
Gradient w.r.t. this module's input, shape (N, in_features).
Note
Must be called after forward() – uses the input cached from that call.
Device-generic: runs on Cpu, Cuda or Hip tensors (GPU-native-kernels Mission 1).
Exceptions
std::logic_errorif forward() has never been called – see campaign_exai_dl_library_adversarial_hardening.md, finding 12.
std::invalid_argumentif grad_output's rank/shape don't match (N, out_features) for the cached N – external boundary, batch-size-mismatch is a new adversarial case this migration introduces (campaign_exai_dl_library_batch_dimension_support).

Implements pulsatrix::Module.

◆ bias()

const Tensor & pulsatrix::LinearModule::bias ( ) const
inline

◆ bias_grad()

const Tensor & pulsatrix::LinearModule::bias_grad ( ) const
inline

◆ compute_device()

std::optional< DeviceType > pulsatrix::LinearModule::compute_device ( ) const
inlineoverridevirtual

Where this layer computes, so forward() rejects an input on another device (FND-8).

Reimplemented from pulsatrix::Module.

◆ detach_ia3()

void pulsatrix::LinearModule::detach_ia3 ( )

Removes the vector. A merged one stays in the weights. No-op without one.

◆ detach_lora()

void pulsatrix::LinearModule::detach_lora ( )

Removes the adapter. A merged one stays in the weight. No-op without one.

◆ forward_impl()

Tensor pulsatrix::LinearModule::forward_impl ( const Tensor &  input)
overrideprotectedvirtual

The actual forward computation. Called by forward() after precondition checks.

Implements pulsatrix::Module.

◆ forward_with_relu()

Tensor pulsatrix::LinearModule::forward_with_relu ( const Tensor &  input,
ReluModule &  relu 
)

relu.forward(forward(input)) with the bias add and the ReLU in one kernel (HIP-6). Both layers end with the cached state the two calls leave, so backward() and LRP are unchanged, and the values are identical. SequentialModule uses it for every LinearModule followed by a ReluModule.

◆ ia3() [1/2]

IA3Adapter * pulsatrix::LinearModule::ia3 ( )
inline

◆ ia3() [2/2]

const IA3Adapter * pulsatrix::LinearModule::ia3 ( ) const
inline

◆ in_features()

int64_t pulsatrix::LinearModule::in_features ( ) const
inline

◆ lora() [1/2]

LoRAAdapter * pulsatrix::LinearModule::lora ( )
inline

The adapter, or nullptr.

◆ lora() [2/2]

const LoRAAdapter * pulsatrix::LinearModule::lora ( ) const
inline

◆ merge_ia3()

void pulsatrix::LinearModule::merge_ia3 ( )

Folds l into the weights (and, for output scaling, the bias), as PEFT's merge() does.

Exceptions
std::logic_errorwithout an IA³ vector, or if it is merged or disabled.

◆ merge_lora()

void pulsatrix::LinearModule::merge_lora ( )

Adds the update into the weight, W += (alpha / r) A B, so forward() runs at the base layer's cost. While merged, A and B get no gradients.

Exceptions
std::logic_errorwithout an adapter, if it is already merged or if it is disabled.

◆ named_parameters()

std::vector< NamedParamRef > pulsatrix::LinearModule::named_parameters ( )
overridevirtual

weight, bias (unless built without one), then a LoRA adapter's lora_A.weight and lora_B.weight, or an IA³ vector's ia3_l.

Reimplemented from pulsatrix::Module.

◆ op_type()

OpType pulsatrix::LinearModule::op_type ( ) const
inlineoverridevirtual

Linear per charter's closed OpType set.

Implements pulsatrix::Module.

◆ operator=()

LinearModule & pulsatrix::LinearModule::operator= ( LinearModule &&  )
noexcept

◆ out_features()

int64_t pulsatrix::LinearModule::out_features ( ) const
inline

◆ propagate_relevance()

Tensor pulsatrix::LinearModule::propagate_relevance ( const Tensor &  relevance_out,
const LRPRuleConfig &  config 
)
overridevirtual

Epsilon-rule LRP relevance propagation (Bach et al. 2015), applied independently per example in the batch.

Parameters
relevance_outRelevance at this module's output. Must be (N, out_features), matching the cached forward() batch size.
configSelects epsilon. Larger epsilon trades a small amount of conservation for numerical stability when a pre-bias output is near zero.
Returns
Relevance at this module's input, shape (N, in_features).
Note
Uses the pre-bias linear output (x @ W, not x @ W + b) as z_j – bias has no associated input feature to redistribute relevance to, so it is excluded from the rule entirely rather than approximated. This is what makes relevance conservation exact (up to the epsilon stabilizer) rather than merely approximate. Must be called after forward() – uses the cached pre-bias output.
Device-generic: runs on Cpu, Cuda or Hip tensors (GPU-native-kernels Mission 3).
config.rule selects Epsilon (default, as above), Gamma, AlphaBeta or ZBox, with Zennit 1.0.0 semantics (see src/lrp_rules.hpp for the formulas). Epsilon with config.epsilon_bias_in_denominator uses z = xW + b instead (Zennit's Epsilon).
Exceptions
std::logic_errorif forward() has never been called – see campaign_exai_dl_library_adversarial_hardening.md, finding 12.
std::invalid_argumentif config's rule parameters are invalid (AlphaBeta needs alpha, beta >= 0 and alpha - beta == 1).

Implements pulsatrix::Module.

◆ release_activations()

void pulsatrix::LinearModule::release_activations ( )
overridevirtual

Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).

Note
Until the next forward(), backward() and relevance calls are invalid. The default has nothing to free.

Reimplemented from pulsatrix::Module.

◆ set_bias() [1/2]

void pulsatrix::LinearModule::set_bias ( const std::vector< float > &  values)

Vector overload for runtime-sized sources – see Tensor's own vector ctor.

◆ set_bias() [2/2]

void pulsatrix::LinearModule::set_bias ( std::initializer_list< float >  values)

Overwrites the bias buffer – test/initialization use only.

Exceptions
std::logic_errorif this layer was built with use_bias = false.

◆ set_ia3_enabled()

void pulsatrix::LinearModule::set_ia3_enabled ( bool  enabled)

false runs the base layer.

Exceptions
std::logic_errorwithout a vector, or while merged.

◆ set_lora_enabled()

void pulsatrix::LinearModule::set_lora_enabled ( bool  enabled)

false runs the layer as its base layer, x W + b; true adds the update again.

Exceptions
std::logic_errorwithout an adapter, or while it is merged.

◆ set_weight() [1/2]

void pulsatrix::LinearModule::set_weight ( const std::vector< float > &  values)

Vector overload for runtime-sized sources – see Tensor's own vector ctor.

◆ set_weight() [2/2]

void pulsatrix::LinearModule::set_weight ( std::initializer_list< float >  values)

Overwrites the weight buffer – test/initialization use only.

◆ supports_lrp_rule()

bool pulsatrix::LinearModule::supports_lrp_rule ( LRPRule  ) const
inlineoverridevirtual

Implements every LRPRule.

Reimplemented from pulsatrix::Module.

◆ unmerge_ia3()

void pulsatrix::LinearModule::unmerge_ia3 ( )

Divides it back out by l + 1e-8, as PEFT's unmerge() does.

Exceptions
std::logic_errorwithout a merged vector.

◆ unmerge_lora()

void pulsatrix::LinearModule::unmerge_lora ( )

Subtracts the update again. W comes back up to float32 rounding, as with PEFT.

Exceptions
std::logic_errorwithout a merged adapter.

◆ uses_bias()

bool pulsatrix::LinearModule::uses_bias ( ) const
inline

False when built with use_bias = false: bias() is then all zeros.

◆ weight()

const Tensor & pulsatrix::LinearModule::weight ( ) const
inline

◆ weight_grad()

const Tensor & pulsatrix::LinearModule::weight_grad ( ) const
inline

The documentation for this class was generated from the following file: