pulsatrix 1.1.0
Loading...
Searching...
No Matches
loreft.hpp
Go to the documentation of this file.
1
17#pragma once
18
19#include <cstdint>
20#include <string>
21#include <vector>
22
24
25namespace pulsatrix {
26
30 int64_t rank = 4;
32 std::vector<int64_t> layers;
35 int64_t first = 1;
36 int64_t last = 1;
38 uint64_t seed = 0;
39};
40
53public:
60 LoReFTIntervention(int64_t d_model, int64_t rank, int64_t first, int64_t last, DeviceBackend* backend, uint64_t seed);
61
62 [[nodiscard]] Tensor forward_at(const Tensor& hidden, int64_t start, int64_t prompt_length) override;
64 [[nodiscard]] Tensor backward(const Tensor& grad_output) override;
72 [[nodiscard]] Tensor propagate_relevance(const Tensor& relevance_out, const LRPRuleConfig& config) override;
73 [[nodiscard]] bool supports_lrp_rule(LRPRule) const override { return true; }
74 [[nodiscard]] OpType op_type() const override { return OpType::Linear; }
75 [[nodiscard]] std::vector<NamedParamRef> named_parameters() override;
76 [[nodiscard]] std::optional<DeviceType> compute_device() const override { return backend_->device(); }
77 void release_activations() override;
78
79 [[nodiscard]] int64_t rank() const { return rank_; }
80 [[nodiscard]] int64_t first() const { return first_; }
81 [[nodiscard]] int64_t last() const { return last_; }
82 [[nodiscard]] const Tensor& rotation() const { return r_; }
83 [[nodiscard]] const Tensor& source_weight() const { return w_; }
84 [[nodiscard]] const Tensor& source_bias() const { return b_; }
86 void set_parameters(const std::vector<float>& rotation, const std::vector<float>& source_weight, const std::vector<float>& source_bias);
90 [[nodiscard]] double orthonormality_error() const;
91
93 [[nodiscard]] const std::vector<int64_t>& last_positions() const { return last_rows_; }
96 [[nodiscard]] const Tensor& last_edit_relevance() const;
97
98private:
99 int64_t d_;
100 int64_t rank_;
101 int64_t first_;
102 int64_t last_;
103 DeviceBackend* backend_;
104 Tensor r_, w_, b_, r_grad_, w_grad_, b_grad_;
105 std::vector<int64_t> last_rows_;
106 Tensor last_indices_; // last_rows_ as floats, for gather_rows/scatter_add_rows
107 Tensor last_selected_; // h at the edited rows, (S, d)
108 Tensor last_coords_; // u = h (W^T - R^T) + b, (S, r)
109 Shape last_shape_ = Shape({0});
110 Tensor last_edit_relevance_;
111};
112
120std::vector<LoReFTIntervention*> ApplyLoReFT(CausalLM& model, const LoReFTConfig& config = {});
121
125
132void SaveLoReFT(CausalLM& model, const std::string& path);
133
141std::vector<LoReFTIntervention*> LoadLoReFT(CausalLM& model, const std::string& path);
142
143} // namespace pulsatrix
A Llama-family decoder-only language model built from a Hugging Face config, and loading its Hugging ...
Token embedding, num_hidden_layers pre-norm transformer blocks, a final RMSNorm and a language-model ...
Definition causal_lm.hpp:82
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
virtual DeviceType device() const noexcept=0
Which device this backend's buffers reside on.
One block's LoReFT edit. Parameters, named as pyreft's state dict:
Definition loreft.hpp:52
int64_t rank() const
Definition loreft.hpp:79
void set_parameters(const std::vector< float > &rotation, const std::vector< float > &source_weight, const std::vector< float > &source_bias)
Replaces R^T (d, r), W^T (d, r) and b (r), row-major.
std::vector< NamedParamRef > named_parameters() override
This module's trainable parameters, each with its hierarchical name – the one place a module declares...
const Tensor & rotation() const
Definition loreft.hpp:82
void release_activations() override
Frees the tensors forward() kept for backward() and the relevance passes; containers release their ch...
const std::vector< int64_t > & last_positions() const
The positions the last forward edited, as row indices into (N * L).
Definition loreft.hpp:93
LoReFTIntervention(int64_t d_model, int64_t rank, int64_t first, int64_t last, DeviceBackend *backend, uint64_t seed)
R orthonormal (QR of a Gaussian matrix), and W and b as an nn.Linear(d, r) starts,...
Tensor backward(const Tensor &grad_output) override
The gradient with respect to the hidden states, accumulating R's, W's and b's.
void orthonormalize()
Makes R's rows orthonormal again (modified Gram-Schmidt in double, signs kept).
const Tensor & source_bias() const
Definition loreft.hpp:84
Tensor propagate_relevance(const Tensor &relevance_out, const LRPRuleConfig &config) override
Identity at the positions it doesn't edit. At the ones it does, the edit is the affine map h (I + (W^...
int64_t first() const
Definition loreft.hpp:80
OpType op_type() const override
This module's operation-category tag, for ComputationGraph node tagging.
Definition loreft.hpp:74
std::optional< DeviceType > compute_device() const override
The device this module computes on, so forward() can reject an input on another device before any ker...
Definition loreft.hpp:76
const Tensor & last_edit_relevance() const
The last relevance pass's part carried by the edit, (N, L, d), zero at unedited positions.
double orthonormality_error() const
The largest |R R^T - I| entry: 0 for an orthonormal R.
int64_t last() const
Definition loreft.hpp:81
const Tensor & source_weight() const
Definition loreft.hpp:83
Tensor forward_at(const Tensor &hidden, int64_t start, int64_t prompt_length) override
The edit for hidden states covering positions [start, start + L) of a sequence whose prompt is prompt...
bool supports_lrp_rule(LRPRule) const override
Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::...
Definition loreft.hpp:73
A trainable edit of the residual stream after a transformer block (TRN-9: LoReFT),...
Definition causal_lm.hpp:53
An N-dimensional shape. A plain aggregate of dimensions with no invariant beyond "non-negative dimens...
Definition shape.hpp:24
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
Definition acquisition_functions.hpp:16
void SaveLoReFT(CausalLM &model, const std::string &path)
Saves the model's LoReFT interventions as safetensors: per layer i, pyreft's state dict under layer....
std::vector< LoReFTIntervention * > ApplyLoReFT(CausalLM &model, const LoReFTConfig &config={})
Installs a LoReFTIntervention after each of config.layers' blocks and freezes the rest of the model: ...
OpType
The op-type tag a Node carries. Charter Part 2 §3: nodes are tagged by a small closed set of op types...
Definition op_type.hpp:19
void OrthonormalizeLoReFT(CausalLM &model)
orthonormalize() on every LoReFT intervention of the model: call it after each optimizer step.
LRPRule
The LRP rule family a module applies. Semantics follow Zennit 1.0.0 exactly (Anders et al....
Definition lrp_rule_config.hpp:21
std::vector< LoReFTIntervention * > LoadLoReFT(CausalLM &model, const std::string &path)
Installs the interventions in a file SaveLoReFT() wrote, or one built from pyreft's per-intervention ...
Configuration for LRP relevance propagation: which rule a module applies and its hyperparameters....
Definition lrp_rule_config.hpp:72
Where and how large LoReFT's interventions are.
Definition loreft.hpp:28
int64_t last
Definition loreft.hpp:36
int64_t first
Edit the first first and the last last positions of each prompt: pyreft's "f<first>+l<last>"....
Definition loreft.hpp:35
uint64_t seed
Seeds R, W and b, the same on every platform.
Definition loreft.hpp:38
int64_t rank
r, the dimension of the edited subspace.
Definition loreft.hpp:30
std::vector< int64_t > layers
The blocks whose outputs are edited. Empty means every block.
Definition loreft.hpp:32