pulsatrix 1.1.0
Loading...
Searching...
No Matches
lora.hpp File Reference

LoRA (Hu et al., 2021): fine-tune a frozen model through a low-rank update on each of its Linear layers, merge it into the weights or take it out again, and save and load it as a PEFT adapter. More...

#include <cstdint>
#include <string>
#include <utility>
#include <vector>
#include "pulsatrix/causal_lm.hpp"
#include "pulsatrix/dropout_module.hpp"
#include "pulsatrix/module.hpp"
#include "pulsatrix/tensor.hpp"
Include dependency graph for lora.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::LoRAConfig
 LoRA's settings. The defaults are PEFT's LoraConfig's. More...
 
class  pulsatrix::LoRAAdapter
 One layer's low-rank update: A (in, r) and B (r, out), stored like LinearModule's weight, with x A B as the update. PEFT's lora_A and lora_B are the transposes, (r, in) and (out, r); SaveLoraAdapter() and LoadLoraAdapter() convert. More...
 
struct  pulsatrix::LoraLayerRelevance
 One adapted layer's share of a relevance pass (TRN-8). More...
 

Namespaces

namespace  pulsatrix
 

Functions

std::vector< std::string > pulsatrix::ApplyLora (Module &model, const LoRAConfig &config={})
 Attaches a LoRA adapter to each LinearModule that config.target_modules selects, then freezes everything else: only the adapters train, as with PEFT's get_peft_model().
 
std::vector< std::pair< std::string, LinearModule * > > pulsatrix::LoraLayers (Module &model)
 The model's LinearModules that have an adapter, by name, in the model's order.
 
std::vector< LoraLayerRelevance > pulsatrix::LoraRelevanceSplit (Module &model)
 How much of each adapted layer's relevance went through its adapter in the model's last relevance pass (LoRARelevance::Split, the default), in the model's order: where, and for which inputs, the fine-tuning changed the explanation.
 
void pulsatrix::MergeLora (Module &model)
 Merges every adapter into its base weight (LinearModule::merge_lora()).
 
void pulsatrix::UnmergeLora (Module &model)
 Takes every merged adapter out of its base weight again.
 
void pulsatrix::SetLoraEnabled (Module &model, bool enabled)
 Turns every adapter on or off, so one model runs as the base or as the fine-tune.
 
void pulsatrix::RemoveLora (Module &model)
 Removes every adapter. Merged ones stay in the weights (PEFT's merge_and_unload() is MergeLora() then this); unmerged ones are dropped (PEFT's unload()).
 
void pulsatrix::SaveLoraAdapter (Module &model, const std::string &directory, const std::vector< WeightMapping > &mapping={})
 Writes the model's adapters as a PEFT adapter directory: adapter_model.safetensors and adapter_config.json, which PEFT's PeftModel.from_pretrained() loads onto the Hugging Face model.
 
void pulsatrix::LoadLoraAdapter (Module &model, const std::string &directory, const std::vector< WeightMapping > &mapping={})
 Reads a PEFT LoRA adapter directory onto the model: attaches an adapter to each layer it has weights for, with r from their shapes and lora_alpha and lora_dropout from adapter_config.json, and freezes the rest as ApplyLora() does.
 

Detailed Description

LoRA (Hu et al., 2021): fine-tune a frozen model through a low-rank update on each of its Linear layers, merge it into the weights or take it out again, and save and load it as a PEFT adapter.

A LoRA layer computes y = x W + b + (alpha / r) x A B, with the base weight W frozen and only A (in, r) and B (r, out) trained. B starts at zero, so the adapted model starts out as the base model. The adapter lives inside a LinearModule (LinearModule::attach_lora()), because the projections of every model in pulsatrix are LinearModule members: ApplyLora() finds them by name and attaches one to each, and the model trains, generates and explains as before.

Use it as "LoRA Without Regret" (Schulman and Thinking Machines Lab, 2025) recommends: on every Linear layer, MLP included (ApplyLora()'s default), with a learning rate about 10 times the one full fine-tuning would use.