Rotary Position Embedding (RoPE, Su et al. 2021): a fixed, non-learnable, position-dependent rotation of each adjacent feature pair of a Q/K-shaped tensor.
More...
#include <rope_module.hpp>
|
| | RoPEModule (int64_t head_dim, DeviceBackend *backend, float base=10000.0f) |
| | Constructs a RoPE module.
|
| |
| Tensor | backward (const Tensor &grad_output) override |
| | Computes the gradient w.r.t. this module's input.
|
| |
| OpType | op_type () const override |
| | Elementwise per the charter's closed OpType set.
|
| |
| Tensor | propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override |
| | Standard weighted-connection epsilon-rule LRP for this fixed linear map.
|
| |
| std::optional< DeviceType > | compute_device () const override |
| | Where this layer computes, so forward() rejects an input on another device (FND-8).
|
| |
| virtual | ~Module ()=default |
| |
| Tensor | forward (const Tensor &input) |
| | Runs this module's forward computation.
|
| |
| std::pair< Tensor, NodeId > | forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd) |
| | Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
|
| |
| virtual bool | supports_lrp_rule (LRPRule rule) const |
| | Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
|
| |
| virtual std::vector< NamedParamRef > | named_parameters () |
| | This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).
|
| |
| virtual std::vector< ParamRef > | parameters () |
| | This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
|
| |
| void | set_requires_grad (bool requires_grad, const std::string &prefix="") |
| | Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
|
| |
| virtual void | set_training (bool training) |
| | Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
|
| |
| bool | is_training () const |
| | Whether this module is currently in training mode.
|
| |
Rotary Position Embedding (RoPE, Su et al. 2021): a fixed, non-learnable, position-dependent rotation of each adjacent feature pair of a Q/K-shaped tensor.
For a feature vector of even dimension head_dim, split into pairs (x[2i], x[2i+1]) for i = 0 .. head_dim/2 - 1. At sequence position pos the rotation angle is theta_i = pos * base^(-2i/head_dim) (base = 10000 by convention), and y[2i] = x[2i]*cos(theta_i) - x[2i+1]*sin(theta_i) y[2i+1] = x[2i]*sin(theta_i) + x[2i+1]*cos(theta_i).
- Note
- Shape convention – rank-agnostic over leading dims, last two axes are
(L, head_dim): input shape (..., L, head_dim) is treated as num_matrices = numel / (L * head_dim) independent (L, head_dim) slices. The position index comes from the L axis (restarting at 0 in every slice) and the feature-pair index from the head_dim axis. This covers (N, L, head_dim) directly as well as a future MultiHeadAttentionModule's (N, num_heads, L, head_dim) with no change – the same rank-agnostic reasoning SoftmaxModule applies to its last axis, one axis deeper.
-
No learnable parameters (
parameters() returns empty, matching ReluModule / SoftmaxModule). Nothing is precomputed across calls either – cos/sin are evaluated directly from pos/i inside the forward loop; this module has no state worth caching an angle table for.
◆ RoPEModule()
| pulsatrix::RoPEModule::RoPEModule |
( |
int64_t |
head_dim, |
|
|
DeviceBackend * |
backend, |
|
|
float |
base = 10000.0f |
|
) |
| |
Constructs a RoPE module.
- Parameters
-
| head_dim | Size of the final (feature) axis. Must be positive and even – the rotation acts on adjacent pairs, so an odd dimension has no valid pairing. |
| backend | Backend to allocate through. Not owned; must outlive this module. |
| base | Frequency base of the geometric angle schedule; 10000.0 is the standard RoPE convention. |
- Exceptions
-
| std::invalid_argument | if head_dim <= 0 or head_dim is odd – external boundary (constructor arguments can originate from Phase 5's Python bindings with no upstream validation), same convention as every other module's constructor argument checks. |
◆ backward()
| Tensor pulsatrix::RoPEModule::backward |
( |
const Tensor & |
grad_output | ) |
|
|
overridevirtual |
Computes the gradient w.r.t. this module's input.
- Parameters
-
| grad_output | Gradient w.r.t. this module's output. Must match the shape of the most recent forward() call's input. |
- Returns
- Gradient w.r.t. this module's input: the inverse rotation applied to grad_output (the rotation matrix is orthogonal, so
R^{-1} = R^T, which is the forward formula with the sin terms' signs swapped): grad_x[2i] = grad_y[2i]*cos(theta_i) + grad_y[2i+1]*sin(theta_i) grad_x[2i+1] = -grad_y[2i]*sin(theta_i) + grad_y[2i+1]*cos(theta_i). Verified against central finite differences in rope_module_test.cpp, not merely asserted from the orthogonality argument.
- Exceptions
-
| std::logic_error | if called before any forward(). |
| std::invalid_argument | if grad_output's shape differs from the cached forward shape. |
- Note
- Device-generic: runs on Cpu, Cuda or Hip tensors (GPU-native-kernels Mission 2).
Implements pulsatrix::Module.
◆ compute_device()
| std::optional< DeviceType > pulsatrix::RoPEModule::compute_device |
( |
| ) |
const |
|
inlineoverridevirtual |
◆ forward_impl()
| Tensor pulsatrix::RoPEModule::forward_impl |
( |
const Tensor & |
input | ) |
|
|
overrideprotectedvirtual |
Applies the per-position pair rotation.
- Parameters
-
| input | Input tensor of shape (..., L, head_dim). Must be rank >= 2 with a final dimension equal to head_dim; any device. |
- Returns
- The rotated tensor, same shape as input.
- Exceptions
-
| std::invalid_argument | if input.rank() < 2 or its last dimension != head_dim. |
Implements pulsatrix::Module.
◆ op_type()
| OpType pulsatrix::RoPEModule::op_type |
( |
| ) |
const |
|
inlineoverridevirtual |
Elementwise per the charter's closed OpType set.
- Note
- Justification for reusing
Elementwise rather than adding an enum value (the enum grows only for genuinely new operation categories): RoPE is a fixed, non-learnable, input-shape-preserving transform whose output element depends only on its own feature pair at its own position – a structured elementwise-*pair* map, strictly local in every axis. It is not Linear (no learnable weight matrix, no contraction across features), not Activation (not a pointwise nonlinearity), not Normalization/Reduction (no statistic is computed over any axis), and not Embedding (no table lookup – the position enters as an angle, not an index into learned vectors). Elementwise is the closest real fit: a shape-preserving, weight-free, local map, which is exactly what a graph query for "which nodes are cheap local transforms" wants this node to answer.
Implements pulsatrix::Module.
◆ propagate_relevance()
Standard weighted-connection epsilon-rule LRP for this fixed linear map.
Each output component of a pair is a two-term weighted sum of the same input pair's two components with fixed cos/sin coefficients, so the epsilon-rule mechanics are identical to RNNModule's two-weighted-source case, just with head_dim-local pairing instead of an across-timestep source. Per pair i at position pos:
R_y[2i] is split across x[2i] (weight cos) and x[2i+1] (weight -sin), proportional to their weighted contributions over y[2i] + eps*sign(y[2i]);R_y[2i+1] is split across x[2i] (weight sin) and x[2i+1] (weight cos) over y[2i+1] + eps*sign(y[2i+1]).
- Parameters
-
| relevance_out | Relevance at this module's output. Must match forward()'s shape. |
| config | Supplies the epsilon stabilizer. |
- Returns
- Relevance at this module's input.
- Note
- Both contributions are summed into each of
R_x[2i]/R_x[2i+1] – every x-component receives relevance from both output components of its pair (structurally analogous to GRUModule's two-path accumulator, simpler because both paths land within one forward() call). The implementation zero-fills the result and uses += for all four writes; an overwrite there would silently drop half the relevance and break conservation.
-
Unlike SoftmaxModule's AttnLRP Eq. 13, this rule does conserve relevance up to the epsilon stabilizer (the map is linear and bias-free, so each pair's two redistributions each sum to their own source relevance times
y/(y + eps*sign(y))). Covered by tests/lrp_conservation_test.cpp's AllModuleTypeCases() entry and by a per-pair conservation test.
- Exceptions
-
| std::logic_error | if called before any forward(). |
| std::invalid_argument | if relevance_out's shape differs from the cached forward shape. |
- Note
- Device-generic: runs on Cpu, Cuda or Hip tensors (GPU-native-kernels Mission 3).
Implements pulsatrix::Module.
The documentation for this class was generated from the following file:
Public Member Functions inherited from pulsatrix::Module
1.9.8