|
pulsatrix 1.1.0
|
IA³ (Liu et al., 2022): fine-tune a frozen model by learning one rescaling vector per adapted Linear layer, on its output (keys, values) or its input (the MLP's down projection), and save and load it as a PEFT adapter. More...
#include <string>#include <utility>#include <vector>#include "pulsatrix/causal_lm.hpp"#include "pulsatrix/module.hpp"#include "pulsatrix/tensor.hpp"
Go to the source code of this file.
Classes | |
| struct | pulsatrix::IA3Config |
| Which layers IA³ rescales, PEFT style (see LoRAConfig::target_modules for the matching). More... | |
| class | pulsatrix::IA3Adapter |
One layer's rescaling vector l: (out) for an output-scaling layer, (in) for a feedforward one. LinearModule owns and drives it; read it through LinearModule::ia3(). More... | |
Namespaces | |
| namespace | pulsatrix |
Functions | |
| std::vector< std::string > | pulsatrix::ApplyIA3 (Module &model, const IA3Config &config={}) |
| Attaches an IA³ vector to each LinearModule config selects, then freezes everything else, as PEFT's get_peft_model() does. | |
| std::vector< std::pair< std::string, LinearModule * > > | pulsatrix::IA3Layers (Module &model) |
| The model's LinearModules with an IA³ vector, by name, in the model's order. | |
| void | pulsatrix::MergeIA3 (Module &model) |
| Folds every vector into its layer's weights (and an output-scaling layer's bias). | |
| void | pulsatrix::UnmergeIA3 (Module &model) |
| Divides it back out, as PEFT does. Entries of l within 1e-8 of 0 can't be undone exactly. | |
| void | pulsatrix::SetIA3Enabled (Module &model, bool enabled) |
| Turns every vector on or off. | |
| void | pulsatrix::RemoveIA3 (Module &model) |
| Removes every vector; merged ones stay in the weights. | |
| void | pulsatrix::SaveIA3Adapter (Module &model, const std::string &directory, const std::vector< WeightMapping > &mapping={}) |
Writes the model's IA³ vectors as a PEFT adapter directory (adapter_model.safetensors with base_model.model.<module>.ia3_l, (out, 1) or (1, in), and adapter_config.json). | |
| void | pulsatrix::LoadIA3Adapter (Module &model, const std::string &directory, const std::vector< WeightMapping > &mapping={}) |
| Reads a PEFT IA³ adapter directory onto the model, telling feedforward layers by the vectors' shapes, and freezes the rest of the model. | |
IA³ (Liu et al., 2022): fine-tune a frozen model by learning one rescaling vector per adapted Linear layer, on its output (keys, values) or its input (the MLP's down projection), and save and load it as a PEFT adapter.
An IA³ layer computes y = (x W + b) ⊙ l, or, for a feedforward layer, y = (x ⊙ l) W + b. Each vector starts at 1, so the adapted model starts as the base model, and it has one value per channel: about 58 thousand values for SmolLM2-135M on the default layers. Either form is the base layer with rescaled weights, W diag(l) or diag(l) W, so the fine-tune can be read off directly (which channels it amplified, which it switched off), and LRP runs on the rescaled layer exactly.