|
pulsatrix 1.1.0
|
LoRA (Hu et al., 2021): fine-tune a frozen model through a low-rank update on each of its Linear layers, merge it into the weights or take it out again, and save and load it as a PEFT adapter. More...
#include <cstdint>#include <string>#include <utility>#include <vector>#include "pulsatrix/causal_lm.hpp"#include "pulsatrix/dropout_module.hpp"#include "pulsatrix/module.hpp"#include "pulsatrix/tensor.hpp"
Go to the source code of this file.
Classes | |
| struct | pulsatrix::LoRAConfig |
LoRA's settings. The defaults are PEFT's LoraConfig's. More... | |
| class | pulsatrix::LoRAAdapter |
One layer's low-rank update: A (in, r) and B (r, out), stored like LinearModule's weight, with x A B as the update. PEFT's lora_A and lora_B are the transposes, (r, in) and (out, r); SaveLoraAdapter() and LoadLoraAdapter() convert. More... | |
| struct | pulsatrix::LoraLayerRelevance |
| One adapted layer's share of a relevance pass (TRN-8). More... | |
Namespaces | |
| namespace | pulsatrix |
Functions | |
| std::vector< std::string > | pulsatrix::ApplyLora (Module &model, const LoRAConfig &config={}) |
| Attaches a LoRA adapter to each LinearModule that config.target_modules selects, then freezes everything else: only the adapters train, as with PEFT's get_peft_model(). | |
| std::vector< std::pair< std::string, LinearModule * > > | pulsatrix::LoraLayers (Module &model) |
| The model's LinearModules that have an adapter, by name, in the model's order. | |
| std::vector< LoraLayerRelevance > | pulsatrix::LoraRelevanceSplit (Module &model) |
| How much of each adapted layer's relevance went through its adapter in the model's last relevance pass (LoRARelevance::Split, the default), in the model's order: where, and for which inputs, the fine-tuning changed the explanation. | |
| void | pulsatrix::MergeLora (Module &model) |
| Merges every adapter into its base weight (LinearModule::merge_lora()). | |
| void | pulsatrix::UnmergeLora (Module &model) |
| Takes every merged adapter out of its base weight again. | |
| void | pulsatrix::SetLoraEnabled (Module &model, bool enabled) |
| Turns every adapter on or off, so one model runs as the base or as the fine-tune. | |
| void | pulsatrix::RemoveLora (Module &model) |
| Removes every adapter. Merged ones stay in the weights (PEFT's merge_and_unload() is MergeLora() then this); unmerged ones are dropped (PEFT's unload()). | |
| void | pulsatrix::SaveLoraAdapter (Module &model, const std::string &directory, const std::vector< WeightMapping > &mapping={}) |
Writes the model's adapters as a PEFT adapter directory: adapter_model.safetensors and adapter_config.json, which PEFT's PeftModel.from_pretrained() loads onto the Hugging Face model. | |
| void | pulsatrix::LoadLoraAdapter (Module &model, const std::string &directory, const std::vector< WeightMapping > &mapping={}) |
Reads a PEFT LoRA adapter directory onto the model: attaches an adapter to each layer it has weights for, with r from their shapes and lora_alpha and lora_dropout from adapter_config.json, and freezes the rest as ApplyLora() does. | |
LoRA (Hu et al., 2021): fine-tune a frozen model through a low-rank update on each of its Linear layers, merge it into the weights or take it out again, and save and load it as a PEFT adapter.
A LoRA layer computes y = x W + b + (alpha / r) x A B, with the base weight W frozen and only A (in, r) and B (r, out) trained. B starts at zero, so the adapted model starts out as the base model. The adapter lives inside a LinearModule (LinearModule::attach_lora()), because the projections of every model in pulsatrix are LinearModule members: ApplyLora() finds them by name and attaches one to each, and the model trains, generates and explains as before.
Use it as "LoRA Without Regret" (Schulman and Thinking Machines Lab, 2025) recommends: on every Linear layer, MLP included (ApplyLora()'s default), with a learning rate about 10 times the one full fine-tuning would use.