y = x @ W + b, batched (x is (N, in_features), y is (N, out_features)) – migrated from the original unbatched (rank-1) scope by campaign_exai_dl_library_batch_dimension_support (breaking migration to always-batched; a single example is N=1, not a structurally different case).
More...
|
| | LinearModule (int64_t in_features, int64_t out_features, DeviceBackend *backend, DeviceType device, bool use_bias=true) |
| | Constructs a linear layer with zero-initialized weight/bias.
|
| |
| | LinearModule (int64_t in_features, int64_t out_features, DeviceBackend *backend, bool use_bias=true) |
| | As above, on backend's own device (backend->device()).
|
| |
| | ~LinearModule () override |
| |
| | LinearModule (LinearModule &&) noexcept |
| |
| LinearModule & | operator= (LinearModule &&) noexcept |
| |
| void | attach_lora (const LoRAConfig &config) |
| | Adds a LoRA adapter (TRN-7, lora.hpp): from now on the layer computes x W + b + (alpha / r) x A B. A is initialized as PEFT does and B is zero, so the output doesn't change until B trains. The base weight isn't frozen here; ApplyLora() freezes it.
|
| |
| void | detach_lora () |
| | Removes the adapter. A merged one stays in the weight. No-op without one.
|
| |
| LoRAAdapter * | lora () |
| | The adapter, or nullptr.
|
| |
| const LoRAAdapter * | lora () const |
| |
| void | merge_lora () |
| | Adds the update into the weight, W += (alpha / r) A B, so forward() runs at the base layer's cost. While merged, A and B get no gradients.
|
| |
| void | unmerge_lora () |
| | Subtracts the update again. W comes back up to float32 rounding, as with PEFT.
|
| |
| void | set_lora_enabled (bool enabled) |
| | false runs the layer as its base layer, x W + b; true adds the update again.
|
| |
| void | attach_ia3 (bool feedforward) |
| | Adds an IA³ rescaling vector l, all ones (TRN-10, ia3.hpp): the layer then computes (x W + b) ⊙ l, or (x ⊙ l) W + b when feedforward. l joins named_parameters() as ia3_l; backward() trains it; propagate_relevance() runs on the rescaled layer, W diag(l) with bias b ⊙ l, or diag(l) W.
|
| |
| void | detach_ia3 () |
| | Removes the vector. A merged one stays in the weights. No-op without one.
|
| |
| IA3Adapter * | ia3 () |
| |
| const IA3Adapter * | ia3 () const |
| |
| void | merge_ia3 () |
| | Folds l into the weights (and, for output scaling, the bias), as PEFT's merge() does.
|
| |
| void | unmerge_ia3 () |
| | Divides it back out by l + 1e-8, as PEFT's unmerge() does.
|
| |
| void | set_ia3_enabled (bool enabled) |
| | false runs the base layer.
|
| |
| Tensor | backward (const Tensor &grad_output) override |
| | Computes the gradient w.r.t. this module's input, and accumulates the weight/bias gradients internally (summed across the batch).
|
| |
| OpType | op_type () const override |
| | Linear per charter's closed OpType set.
|
| |
| void | set_weight (std::initializer_list< float > values) |
| | Overwrites the weight buffer – test/initialization use only.
|
| |
| void | set_bias (std::initializer_list< float > values) |
| | Overwrites the bias buffer – test/initialization use only.
|
| |
| void | set_weight (const std::vector< float > &values) |
| | Vector overload for runtime-sized sources – see Tensor's own vector ctor.
|
| |
| void | set_bias (const std::vector< float > &values) |
| | Vector overload for runtime-sized sources – see Tensor's own vector ctor.
|
| |
| int64_t | in_features () const |
| |
| int64_t | out_features () const |
| |
| bool | uses_bias () const |
| | False when built with use_bias = false: bias() is then all zeros.
|
| |
| const Tensor & | weight () const |
| |
| const Tensor & | bias () const |
| |
| const Tensor & | weight_grad () const |
| |
| const Tensor & | bias_grad () const |
| |
| Tensor | propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override |
| | Epsilon-rule LRP relevance propagation (Bach et al. 2015), applied independently per example in the batch.
|
| |
| Tensor | forward_with_relu (const Tensor &input, ReluModule &relu) |
| | relu.forward(forward(input)) with the bias add and the ReLU in one kernel (HIP-6). Both layers end with the cached state the two calls leave, so backward() and LRP are unchanged, and the values are identical. SequentialModule uses it for every LinearModule followed by a ReluModule.
|
| |
| bool | supports_lrp_rule (LRPRule) const override |
| | Implements every LRPRule.
|
| |
| std::vector< NamedParamRef > | named_parameters () override |
| | weight, bias (unless built without one), then a LoRA adapter's lora_A.weight and lora_B.weight, or an IA³ vector's ia3_l.
|
| |
| std::optional< DeviceType > | compute_device () const override |
| | Where this layer computes, so forward() rejects an input on another device (FND-8).
|
| |
| void | release_activations () override |
| | Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
|
| |
| virtual | ~Module ()=default |
| |
| Tensor | forward (const Tensor &input) |
| | Runs this module's forward computation.
|
| |
| std::pair< Tensor, NodeId > | forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd) |
| | Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
|
| |
| virtual std::vector< NamedBufferRef > | named_buffers () |
| | This module's buffers: state a checkpoint must save that no optimizer updates.
|
| |
| virtual std::vector< NamedModuleRef > | named_children () |
| | This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
|
| |
| virtual std::vector< ParamRef > | parameters () |
| | This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
|
| |
| void | set_requires_grad (bool requires_grad, const std::string &prefix="") |
| | Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
|
| |
| virtual void | set_training (bool training) |
| | Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
|
| |
| bool | is_training () const |
| | Whether this module is currently in training mode.
|
| |
y = x @ W + b, batched (x is (N, in_features), y is (N, out_features)) – migrated from the original unbatched (rank-1) scope by campaign_exai_dl_library_batch_dimension_support (breaking migration to always-batched; a single example is N=1, not a structurally different case).
- Note
- Weight layout is (in_features, out_features), not the more common (out_features, in_features) PyTorch convention – chosen specifically so forward (x @ W) and the weight-gradient step (X^T @ grad_Y) both use DeviceBackend::gemm directly with no transpose of the weight operand. The input-gradient step (grad_Y @ W^T) and the weight-gradient step (X^T @ grad_Y, the batched sum of outer products) read their transposed operand in place through DeviceBackend::gemm_ex – no transposed copy is built.
-
Weights/biases are owned here as member Tensors, not ComputationGraph nodes. Parameter gradients accumulate via Tensor::accumulate() across backward() calls until something (the optimizer) resets them. Batch-dimension gradient reduction (summing weight/bias gradient contributions across the N examples in a batch) needs no new Tensor primitive – see campaign_exai_dl_library_batch_dimension_support's mission_tensor_shape_foundation.md: weight_grad's reduction falls out of gemm's own k-dimension summation (accumulated in place, beta = 1); bias_grad's is DeviceBackend::column_sums, likewise accumulated in place. Both are device-resident (GPU-native-kernels Mission 1).