pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::RoPEModule Class Reference

Rotary Position Embedding (RoPE, Su et al. 2021): a fixed, non-learnable, position-dependent rotation of each adjacent feature pair of a Q/K-shaped tensor. More...

#include <rope_module.hpp>

Inheritance diagram for pulsatrix::RoPEModule:
Collaboration diagram for pulsatrix::RoPEModule:

Public Member Functions

 RoPEModule (int64_t head_dim, DeviceBackend *backend, float base=10000.0f, RoPELayout layout=RoPELayout::AdjacentPairs)
 Constructs a RoPE module.
 
 RoPEModule (int64_t head_dim, DeviceBackend *backend, std::vector< double > inverse_frequencies, RoPELayout layout=RoPELayout::AdjacentPairs)
 RoPE with explicit per-pair frequencies (LLM-6): pair i turns by pos * inverse_frequencies[i] instead of pos * base^(-2i/head_dim), which is how scaled variants such as Llama 3's are expressed.
 
void set_position_offset (int64_t offset)
 Sets the position of the first row of the L axis, so forward() rotates rows offset, offset + 1, ... instead of 0, 1, ... (LLM-1). Used when a sequence is processed in pieces, as generation with a KV cache does.
 
int64_t position_offset () const
 
RoPELayout layout () const
 
float base () const
 
Tensor backward (const Tensor &grad_output) override
 Computes the gradient w.r.t. this module's input.
 
OpType op_type () const override
 Elementwise per the charter's closed OpType set.
 
Tensor propagate_relevance (const Tensor &relevance_out, const LRPRuleConfig &config) override
 Standard weighted-connection epsilon-rule LRP for this fixed linear map.
 
std::optional< DeviceType > compute_device () const override
 Where this layer computes, so forward() rejects an input on another device (FND-8).
 
void release_activations () override
 Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).
 
- Public Member Functions inherited from pulsatrix::Module
virtual ~Module ()=default
 
Tensor forward (const Tensor &input)
 Runs this module's forward computation.
 
std::pair< Tensor, NodeId > forward_traced (const Tensor &input, NodeId input_node, ComputationGraph &graph, Autograd &autograd)
 Runs forward() while also registering a ComputationGraph node (tagged with this module's op_type(), parented to input_node) and wiring an Autograd backward function that reuses this module's own backward() – the opt-in traced/explainable path, per Phase 2 Mission 0.
 
virtual bool supports_lrp_rule (LRPRule rule) const
 Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::relevance_pass() throw rather than run a module on a rule it does not implement).
 
virtual std::vector< NamedParamRef > named_parameters ()
 This module's trainable parameters, each with its hierarchical name – the one place a module declares its parameters (roadmap FND-1).
 
virtual std::vector< NamedBufferRef > named_buffers ()
 This module's buffers: state a checkpoint must save that no optimizer updates.
 
virtual std::vector< NamedModuleRef > named_children ()
 This module's direct sub-modules, each under the prefix its parameters carry in named_parameters(), in the same order (TRN-7: how ApplyLora() finds a model's Linear layers). Default: none. NamedModules() walks the whole tree.
 
virtual std::vector< ParamRef > parameters ()
 This module's trainable parameters and their gradients, for an optimizer to update uniformly across module types.
 
void set_requires_grad (bool requires_grad, const std::string &prefix="")
 Freezes (false) or unfreezes (true) parameters by name (roadmap FND-2).
 
virtual void set_training (bool training)
 Sets this module's training/eval mode. Defaults to training (matches every mainstream framework's Module default).
 
bool is_training () const
 Whether this module is currently in training mode.
 

Protected Member Functions

Tensor forward_impl (const Tensor &input) override
 Applies the per-position pair rotation.
 

Detailed Description

Rotary Position Embedding (RoPE, Su et al. 2021): a fixed, non-learnable, position-dependent rotation of each adjacent feature pair of a Q/K-shaped tensor.

For a feature vector of even dimension head_dim, split into pairs (x[2i], x[2i+1]) for i = 0 .. head_dim/2 - 1 (RoPELayout::AdjacentPairs; RoPELayout::RotateHalf pairs (x[i], x[i + head_dim/2]) instead, with the same formulas). At sequence position pos the rotation angle is theta_i = pos * base^(-2i/head_dim) (base = 10000 by convention), and y[2i] = x[2i]*cos(theta_i) - x[2i+1]*sin(theta_i) y[2i+1] = x[2i]*sin(theta_i) + x[2i+1]*cos(theta_i).

Note
Shape convention – rank-agnostic over leading dims, last two axes are (L, head_dim): input shape (..., L, head_dim) is treated as num_matrices = numel / (L * head_dim) independent (L, head_dim) slices. The position index comes from the L axis (restarting at 0 in every slice) and the feature-pair index from the head_dim axis. This covers (N, L, head_dim) directly as well as a future MultiHeadAttentionModule's (N, num_heads, L, head_dim) with no change – the same rank-agnostic reasoning SoftmaxModule applies to its last axis, one axis deeper.
No learnable parameters (parameters() returns empty, matching ReluModule / SoftmaxModule). Nothing is precomputed across calls either – cos/sin are evaluated directly from pos/i inside the forward loop; this module has no state worth caching an angle table for.

Constructor & Destructor Documentation

◆ RoPEModule() [1/2]

pulsatrix::RoPEModule::RoPEModule ( int64_t  head_dim,
DeviceBackend *  backend,
float  base = 10000.0f,
RoPELayout  layout = RoPELayout::AdjacentPairs 
)

Constructs a RoPE module.

Parameters
head_dimSize of the final (feature) axis. Must be positive and even – the rotation acts on adjacent pairs, so an odd dimension has no valid pairing.
backendBackend to allocate through. Not owned; must outlive this module.
baseFrequency base of the geometric angle schedule; 10000.0 is the standard RoPE convention. Hugging Face configs call it rope_theta.
layoutWhich features form each rotated pair; see RoPELayout.
Exceptions
std::invalid_argumentif head_dim <= 0 or head_dim is odd – external boundary (constructor arguments can originate from Phase 5's Python bindings with no upstream validation), same convention as every other module's constructor argument checks.

◆ RoPEModule() [2/2]

pulsatrix::RoPEModule::RoPEModule ( int64_t  head_dim,
DeviceBackend *  backend,
std::vector< double >  inverse_frequencies,
RoPELayout  layout = RoPELayout::AdjacentPairs 
)

RoPE with explicit per-pair frequencies (LLM-6): pair i turns by pos * inverse_frequencies[i] instead of pos * base^(-2i/head_dim), which is how scaled variants such as Llama 3's are expressed.

Exceptions
std::invalid_argumentif head_dim is invalid, or there aren't head_dim / 2 finite, positive frequencies.

Member Function Documentation

◆ backward()

Tensor pulsatrix::RoPEModule::backward ( const Tensor &  grad_output)
overridevirtual

Computes the gradient w.r.t. this module's input.

Parameters
grad_outputGradient w.r.t. this module's output. Must match the shape of the most recent forward() call's input.
Returns
Gradient w.r.t. this module's input: the inverse rotation applied to grad_output (the rotation matrix is orthogonal, so R^{-1} = R^T, which is the forward formula with the sin terms' signs swapped): grad_x[2i] = grad_y[2i]*cos(theta_i) + grad_y[2i+1]*sin(theta_i) grad_x[2i+1] = -grad_y[2i]*sin(theta_i) + grad_y[2i+1]*cos(theta_i). Verified against central finite differences in rope_module_test.cpp, not merely asserted from the orthogonality argument.
Exceptions
std::logic_errorif called before any forward().
std::invalid_argumentif grad_output's shape differs from the cached forward shape.
Note
Device-generic: runs on Cpu, Cuda or Hip tensors (GPU-native-kernels Mission 2).

Implements pulsatrix::Module.

◆ base()

float pulsatrix::RoPEModule::base ( ) const
inline

◆ compute_device()

std::optional< DeviceType > pulsatrix::RoPEModule::compute_device ( ) const
inlineoverridevirtual

Where this layer computes, so forward() rejects an input on another device (FND-8).

Reimplemented from pulsatrix::Module.

◆ forward_impl()

Tensor pulsatrix::RoPEModule::forward_impl ( const Tensor &  input)
overrideprotectedvirtual

Applies the per-position pair rotation.

Parameters
inputInput tensor of shape (..., L, head_dim). Must be rank >= 2 with a final dimension equal to head_dim; any device.
Returns
The rotated tensor, same shape as input.
Exceptions
std::invalid_argumentif input.rank() < 2 or its last dimension != head_dim.

Implements pulsatrix::Module.

◆ layout()

RoPELayout pulsatrix::RoPEModule::layout ( ) const
inline

◆ op_type()

OpType pulsatrix::RoPEModule::op_type ( ) const
inlineoverridevirtual

Elementwise per the charter's closed OpType set.

Note
Justification for reusing Elementwise rather than adding an enum value (the enum grows only for genuinely new operation categories): RoPE is a fixed, non-learnable, input-shape-preserving transform whose output element depends only on its own feature pair at its own position – a structured elementwise-*pair* map, strictly local in every axis. It is not Linear (no learnable weight matrix, no contraction across features), not Activation (not a pointwise nonlinearity), not Normalization/Reduction (no statistic is computed over any axis), and not Embedding (no table lookup – the position enters as an angle, not an index into learned vectors). Elementwise is the closest real fit: a shape-preserving, weight-free, local map, which is exactly what a graph query for "which nodes are cheap local transforms" wants this node to answer.

Implements pulsatrix::Module.

◆ position_offset()

int64_t pulsatrix::RoPEModule::position_offset ( ) const
inline

◆ propagate_relevance()

Tensor pulsatrix::RoPEModule::propagate_relevance ( const Tensor &  relevance_out,
const LRPRuleConfig &  config 
)
overridevirtual

Standard weighted-connection epsilon-rule LRP for this fixed linear map.

Each output component of a pair is a two-term weighted sum of the same input pair's two components with fixed cos/sin coefficients, so the epsilon-rule mechanics are identical to RNNModule's two-weighted-source case, just with head_dim-local pairing instead of an across-timestep source. Per pair i at position pos:

  • R_y[2i] is split across x[2i] (weight cos) and x[2i+1] (weight -sin), proportional to their weighted contributions over y[2i] + eps*sign(y[2i]);
  • R_y[2i+1] is split across x[2i] (weight sin) and x[2i+1] (weight cos) over y[2i+1] + eps*sign(y[2i+1]).
Parameters
relevance_outRelevance at this module's output. Must match forward()'s shape.
configSupplies the epsilon stabilizer.
Returns
Relevance at this module's input.
Note
Both contributions are summed into each of R_x[2i]/R_x[2i+1] – every x-component receives relevance from both output components of its pair (structurally analogous to GRUModule's two-path accumulator, simpler because both paths land within one forward() call). The implementation zero-fills the result and uses += for all four writes; an overwrite there would silently drop half the relevance and break conservation.
Unlike SoftmaxModule's AttnLRP Eq. 13, this rule does conserve relevance up to the epsilon stabilizer (the map is linear and bias-free, so each pair's two redistributions each sum to their own source relevance times y/(y + eps*sign(y))). Covered by tests/lrp_conservation_test.cpp's AllModuleTypeCases() entry and by a per-pair conservation test.
Exceptions
std::logic_errorif called before any forward().
std::invalid_argumentif relevance_out's shape differs from the cached forward shape.
Note
Device-generic: runs on Cpu, Cuda or Hip tensors (GPU-native-kernels Mission 3).

Implements pulsatrix::Module.

◆ release_activations()

void pulsatrix::RoPEModule::release_activations ( )
overridevirtual

Frees the tensors forward() kept for backward() and the relevance passes; containers release their children too. Inference that only needs the output calls it after each forward, so a deep model holds one layer's activations at a time (a 1024-token ESM-2 650M pass otherwise keeps over 10 GB of them).

Note
Until the next forward(), backward() and relevance calls are invalid. The default has nothing to free.

Reimplemented from pulsatrix::Module.

◆ set_position_offset()

void pulsatrix::RoPEModule::set_position_offset ( int64_t  offset)

Sets the position of the first row of the L axis, so forward() rotates rows offset, offset + 1, ... instead of 0, 1, ... (LLM-1). Used when a sequence is processed in pieces, as generation with a KV cache does.

Exceptions
std::invalid_argumentif offset is negative.

The documentation for this class was generated from the following file: