|
| | LoReFTIntervention (int64_t d_model, int64_t rank, int64_t first, int64_t last, DeviceBackend *backend, uint64_t seed) |
| | R orthonormal (QR of a Gaussian matrix), and W and b as an nn.Linear(d, r) starts, uniform on ±1/sqrt(d), as pyreft initializes them. Unlike LoRA, the edit isn't zero at the start.
|
| |
| Tensor | forward_at (const Tensor &hidden, int64_t start, int64_t prompt_length) override |
| | The edit for hidden states covering positions [start, start + L) of a sequence whose prompt is prompt_length positions long. forward(x) is forward_at(x, 0, L). Cached generation calls it once on the prompt and then once per new token, each a later start.
|
| |
| Tensor | backward (const Tensor &grad_output) override |
| | The gradient with respect to the hidden states, accumulating R's, W's and b's.
|
| |
| Tensor | propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override |
| | Identity at the positions it doesn't edit. At the ones it does, the edit is the affine map h (I + (W^T - R^T) R) + b R, and the rule runs on its two paths together, the identity and the edit, as for a split LoRA layer (TRN-8). The edit's part is kept in last_edit_relevance(). Every rule is supported; under epsilon the result is the relevance of the map as a whole.
|
| |
| bool | supports_lrp_rule (LRPRule) const override |
| | Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
|
| |
| OpType | op_type () const override |
| | This module's operation-category tag, for ComputationGraph node tagging.
|
| |
| std::vector< NamedParamRef > | named_parameters () override |
| | This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).
|
| |
| std::optional< DeviceType > | compute_device () const override |
| | The device this module computes on, so forward() can reject an input on another device before any kernel sees it (roadmap FND-8).
|
| |
| void | release_activations () override |
| | Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
|
| |
| int64_t | rank () const |
| |
| int64_t | first () const |
| |
| int64_t | last () const |
| |
| const Tensor & | rotation () const |
| |
| const Tensor & | source_weight () const |
| |
| const Tensor & | source_bias () const |
| |
| void | set_parameters (const std::vector< float > &rotation, const std::vector< float > &source_weight, const std::vector< float > &source_bias) |
| | Replaces R^T (d, r), W^T (d, r) and b (r), row-major.
|
| |
| void | orthonormalize () |
| | Makes R's rows orthonormal again (modified Gram-Schmidt in double, signs kept).
|
| |
| double | orthonormality_error () const |
| | The largest |R R^T - I| entry: 0 for an orthonormal R.
|
| |
| const std::vector< int64_t > & | last_positions () const |
| | The positions the last forward edited, as row indices into (N * L).
|
| |
| const Tensor & | last_edit_relevance () const |
| | The last relevance pass's part carried by the edit, (N, L, d), zero at unedited positions.
|
| |
| virtual | ~Module ()=default |
| |
| Tensor | forward (const Tensor &input) |
| | Runs this module's forward computation.
|
| |
| std::pair< Tensor, NodeId > | forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd) |
| | Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
|
| |
| virtual std::vector< NamedBufferRef > | named_buffers () |
| | This module's buffers: state a checkpoint must save that no optimizer updates.
|
| |
| virtual std::vector< NamedModuleRef > | named_children () |
| | This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
|
| |
| virtual std::vector< ParamRef > | parameters () |
| | This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
|
| |
| void | set_requires_grad (bool requires_grad, const std::string &prefix="") |
| | Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
|
| |
| virtual void | set_training (bool training) |
| | Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
|
| |
| bool | is_training () const |
| | Whether this module is currently in training mode.
|
| |
One block's LoReFT edit. Parameters, named as pyreft's state dict:
rotate_layer: R^T, (d, r) with orthonormal columns (pyreft's layout);
learned_source.weight: W^T, (d, r), the transpose of pyreft's (r, d), like every pulsatrix Linear weight;
learned_source.bias: b, (r). - Note
- Gradient steps move R off the orthonormal set; call orthonormalize() after each optimizer step (OrthonormalizeLoReFT() does it for a model). pyreft keeps R orthonormal with torch's orthogonal parametrization instead, so training paths differ, but the edit, its gradient with respect to R and its relevance are the same for the same R.