pulsatrix 1.1.0
Loading...
Searching...
No Matches
ia3.hpp File Reference

IA³ (Liu et al., 2022): fine-tune a frozen model by learning one rescaling vector per adapted Linear layer, on its output (keys, values) or its input (the MLP's down projection), and save and load it as a PEFT adapter. More...

#include <string>
#include <utility>
#include <vector>
#include "pulsatrix/causal_lm.hpp"
#include "pulsatrix/module.hpp"
#include "pulsatrix/tensor.hpp"
Include dependency graph for ia3.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::IA3Config
 Which layers IA³ rescales, PEFT style (see LoRAConfig::target_modules for the matching). More...
 
class  pulsatrix::IA3Adapter
 One layer's rescaling vector l: (out) for an output-scaling layer, (in) for a feedforward one. LinearModule owns and drives it; read it through LinearModule::ia3(). More...
 

Namespaces

namespace  pulsatrix
 

Functions

std::vector< std::string > pulsatrix::ApplyIA3 (Module &model, const IA3Config &config={})
 Attaches an IA³ vector to each LinearModule config selects, then freezes everything else, as PEFT's get_peft_model() does.
 
std::vector< std::pair< std::string, LinearModule * > > pulsatrix::IA3Layers (Module &model)
 The model's LinearModules with an IA³ vector, by name, in the model's order.
 
void pulsatrix::MergeIA3 (Module &model)
 Folds every vector into its layer's weights (and an output-scaling layer's bias).
 
void pulsatrix::UnmergeIA3 (Module &model)
 Divides it back out, as PEFT does. Entries of l within 1e-8 of 0 can't be undone exactly.
 
void pulsatrix::SetIA3Enabled (Module &model, bool enabled)
 Turns every vector on or off.
 
void pulsatrix::RemoveIA3 (Module &model)
 Removes every vector; merged ones stay in the weights.
 
void pulsatrix::SaveIA3Adapter (Module &model, const std::string &directory, const std::vector< WeightMapping > &mapping={})
 Writes the model's IA³ vectors as a PEFT adapter directory (adapter_model.safetensors with base_model.model.<module>.ia3_l, (out, 1) or (1, in), and adapter_config.json).
 
void pulsatrix::LoadIA3Adapter (Module &model, const std::string &directory, const std::vector< WeightMapping > &mapping={})
 Reads a PEFT IA³ adapter directory onto the model, telling feedforward layers by the vectors' shapes, and freezes the rest of the model.
 

Detailed Description

IA³ (Liu et al., 2022): fine-tune a frozen model by learning one rescaling vector per adapted Linear layer, on its output (keys, values) or its input (the MLP's down projection), and save and load it as a PEFT adapter.

An IA³ layer computes y = (x W + b) ⊙ l, or, for a feedforward layer, y = (x ⊙ l) W + b. Each vector starts at 1, so the adapted model starts as the base model, and it has one value per channel: about 58 thousand values for SmolLM2-135M on the default layers. Either form is the base layer with rescaled weights, W diag(l) or diag(l) W, so the fine-tune can be read off directly (which channels it amplified, which it switched off), and LRP runs on the rescaled layer exactly.