|
pulsatrix 1.1.0
|
Rotary Position Embedding (RoPE, Su et al. 2021): a fixed, non-learnable, position-dependent rotation of each adjacent feature pair of a Q/K-shaped tensor. More...
#include <rope_module.hpp>


Public Member Functions | |
| RoPEModule (int64_t head_dim, DeviceBackend *backend, float base=10000.0f, RoPELayout layout=RoPELayout::AdjacentPairs) | |
| Constructs a RoPE module. | |
| RoPEModule (int64_t head_dim, DeviceBackend *backend, std::vector< double > inverse_frequencies, RoPELayout layout=RoPELayout::AdjacentPairs) | |
RoPE with explicit per-pair frequencies (LLM-6): pair i turns by pos * inverse_frequencies[i] instead of pos * base^(-2i/head_dim), which is how scaled variants such as Llama 3's are expressed. | |
| void | set_position_offset (int64_t offset) |
Sets the position of the first row of the L axis, so forward() rotates rows offset, offset + 1, ... instead of 0, 1, ... (LLM-1). Used when a sequence is processed in pieces, as generation with a KV cache does. | |
| int64_t | position_offset () const |
| RoPELayout | layout () const |
| float | base () const |
| Tensor | backward (const Tensor &grad_output) override |
| Computes the gradient w.r.t. this module's input. | |
| OpType | op_type () const override |
| Elementwise per the charter's closed OpType set. | |
| Tensor | propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override |
| Standard weighted-connection epsilon-rule LRP for this fixed linear map. | |
| std::optional< DeviceType > | compute_device () const override |
| Where this layer computes, so forward() rejects an input on another device (FND-8). | |
| void | release_activations () override |
| Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them). | |
Public Member Functions inherited from pulsatrix::Module | |
| virtual | ~Module ()=default |
| Tensor | forward (const Tensor &input) |
| Runs this module's forward computation. | |
| std::pair< Tensor, NodeId > | forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd) |
| Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0. | |
| virtual bool | supports_lrp_rule (LRPRule rule) const |
Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement). | |
| virtual std::vector< NamedParamRef > | named_parameters () |
| This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1). | |
| virtual std::vector< NamedBufferRef > | named_buffers () |
| This module's buffers: state a checkpoint must save that no optimizer updates. | |
| virtual std::vector< NamedModuleRef > | named_children () |
| This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree. | |
| virtual std::vector< ParamRef > | parameters () |
| This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types. | |
| void | set_requires_grad (bool requires_grad, const std::string &prefix="") |
Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2). | |
| virtual void | set_training (bool training) |
| Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default). | |
| bool | is_training () const |
| Whether this module is currently in training mode. | |
Protected Member Functions | |
| Tensor | forward_impl (const Tensor &input) override |
| Applies the per-position pair rotation. | |
Rotary Position Embedding (RoPE, Su et al. 2021): a fixed, non-learnable, position-dependent rotation of each adjacent feature pair of a Q/K-shaped tensor.
For a feature vector of even dimension head_dim, split into pairs (x[2i], x[2i+1]) for i = 0 .. head_dim/2 - 1 (RoPELayout::AdjacentPairs; RoPELayout::RotateHalf pairs (x[i], x[i + head_dim/2]) instead, with the same formulas). At sequence position pos the rotation angle is theta_i = pos * base^(-2i/head_dim) (base = 10000 by convention), and y[2i] = x[2i]*cos(theta_i) - x[2i+1]*sin(theta_i) y[2i+1] = x[2i]*sin(theta_i) + x[2i+1]*cos(theta_i).
(L, head_dim): input shape (..., L, head_dim) is treated as num_matrices = numel / (L * head_dim) independent (L, head_dim) slices. The position index comes from the L axis (restarting at 0 in every slice) and the feature-pair index from the head_dim axis. This covers (N, L, head_dim) directly as well as a future MultiHeadAttentionModule's (N, num_heads, L, head_dim) with no change – the same rank-agnostic reasoning SoftmaxModule applies to its last axis, one axis deeper. parameters() returns empty, matching ReluModule / SoftmaxModule). Nothing is precomputed across calls either – cos/sin are evaluated directly from pos/i inside the forward loop; this module has no state worth caching an angle table for. | pulsatrix::RoPEModule::RoPEModule | ( | int64_t | head_dim, |
| DeviceBackend * | backend, | ||
| float | base = 10000.0f, |
||
| RoPELayout | layout = RoPELayout::AdjacentPairs |
||
| ) |
Constructs a RoPE module.
| head_dim | Size of the final (feature) axis. Must be positive and even – the rotation acts on adjacent pairs, so an odd dimension has no valid pairing. |
| backend | Backend to allocate through. Not owned; must outlive this module. |
| base | Frequency base of the geometric angle schedule; 10000.0 is the standard RoPE convention. Hugging Face configs call it rope_theta. |
| layout | Which features form each rotated pair; see RoPELayout. |
| std::invalid_argument | if head_dim <= 0 or head_dim is odd – external boundary (constructor arguments can originate from Phase 5's Python bindings with no upstream validation), same convention as every other module's constructor argument checks. |
| pulsatrix::RoPEModule::RoPEModule | ( | int64_t | head_dim, |
| DeviceBackend * | backend, | ||
| std::vector< double > | inverse_frequencies, | ||
| RoPELayout | layout = RoPELayout::AdjacentPairs |
||
| ) |
RoPE with explicit per-pair frequencies (LLM-6): pair i turns by pos * inverse_frequencies[i] instead of pos * base^(-2i/head_dim), which is how scaled variants such as Llama 3's are expressed.
| std::invalid_argument | if head_dim is invalid, or there aren't head_dim / 2 finite, positive frequencies. |
Computes the gradient w.r.t. this module's input.
| grad_output | Gradient w.r.t. this module's output. Must match the shape of the most recent forward() call's input. |
R^{-1} = R^T, which is the forward formula with the sin terms' signs swapped): grad_x[2i] = grad_y[2i]*cos(theta_i) + grad_y[2i+1]*sin(theta_i) grad_x[2i+1] = -grad_y[2i]*sin(theta_i) + grad_y[2i+1]*cos(theta_i). Verified against central finite differences in rope_module_test.cpp, not merely asserted from the orthogonality argument. | std::logic_error | if called before any forward(). |
| std::invalid_argument | if grad_output's shape differs from the cached forward shape. |
Implements pulsatrix::Module.
|
inline |
|
inlineoverridevirtual |
Where this layer computes, so forward() rejects an input on another device (FND-8).
Reimplemented from pulsatrix::Module.
Applies the per-position pair rotation.
| input | Input tensor of shape (..., L, head_dim). Must be rank >= 2 with a final dimension equal to head_dim; any device. |
| std::invalid_argument | if input.rank() < 2 or its last dimension != head_dim. |
Implements pulsatrix::Module.
|
inline |
|
inlineoverridevirtual |
Elementwise per the charter's closed OpType set.
Elementwise rather than adding an enum value (the enum grows only for genuinely new operation categories): RoPE is a fixed, non-learnable, input-shape-preserving transform whose output element depends only on its own feature pair at its own position – a structured elementwise-*pair* map, strictly local in every axis. It is not Linear (no learnable weight matrix, no contraction across features), not Activation (not a pointwise nonlinearity), not Normalization/Reduction (no statistic is computed over any axis), and not Embedding (no table lookup – the position enters as an angle, not an index into learned vectors). Elementwise is the closest real fit: a shape-preserving, weight-free, local map, which is exactly what a graph query for "which nodes are cheap local transforms" wants this node to answer. Implements pulsatrix::Module.
|
inline |
|
overridevirtual |
Standard weighted-connection epsilon-rule LRP for this fixed linear map.
Each output component of a pair is a two-term weighted sum of the same input pair's two components with fixed cos/sin coefficients, so the epsilon-rule mechanics are identical to RNNModule's two-weighted-source case, just with head_dim-local pairing instead of an across-timestep source. Per pair i at position pos:
R_y[2i] is split across x[2i] (weight cos) and x[2i+1] (weight -sin), proportional to their weighted contributions over y[2i] + eps*sign(y[2i]);R_y[2i+1] is split across x[2i] (weight sin) and x[2i+1] (weight cos) over y[2i+1] + eps*sign(y[2i+1]).| relevance_out | Relevance at this module's output. Must match forward()'s shape. |
| config | Supplies the epsilon stabilizer. |
R_x[2i]/R_x[2i+1] – every x-component receives relevance from both output components of its pair (structurally analogous to GRUModule's two-path accumulator, simpler because both paths land within one forward() call). The implementation zero-fills the result and uses += for all four writes; an overwrite there would silently drop half the relevance and break conservation. y/(y + eps*sign(y))). Covered by tests/lrp_conservation_test.cpp's AllModuleTypeCases() entry and by a per-pair conservation test. | std::logic_error | if called before any forward(). |
| std::invalid_argument | if relevance_out's shape differs from the cached forward shape. |
Implements pulsatrix::Module.
|
overridevirtual |
Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
Reimplemented from pulsatrix::Module.
| void pulsatrix::RoPEModule::set_position_offset | ( | int64_t | offset | ) |
Sets the position of the first row of the L axis, so forward() rotates rows offset, offset + 1, ... instead of 0, 1, ... (LLM-1). Used when a sequence is processed in pieces, as generation with a KV cache does.
| std::invalid_argument | if offset is negative. |