pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::IA3Config Struct Reference

Which layers IA³ rescales, PEFT style (see LoRAConfig::target_modules for the matching). More...

#include <ia3.hpp>

Public Attributes

std::vector< std::string > target_modules
 The layers to rescale. Empty: the paper's choice, keys, values and the MLP's inner activations: every k_proj, v_proj, and down_proj or fc2. (PEFT's default for Qwen2 is q_proj, v_proj and down_proj; for Llama it is the paper's.)
 
std::vector< std::string > feedforward_modules
 Those whose input is rescaled rather than their output. Each must be among the targets. Empty with empty targets: down_proj and fc2.
 

Detailed Description

Which layers IA³ rescales, PEFT style (see LoRAConfig::target_modules for the matching).

Member Data Documentation

◆ feedforward_modules

std::vector<std::string> pulsatrix::IA3Config::feedforward_modules

Those whose input is rescaled rather than their output. Each must be among the targets. Empty with empty targets: down_proj and fc2.

◆ target_modules

std::vector<std::string> pulsatrix::IA3Config::target_modules

The layers to rescale. Empty: the paper's choice, keys, values and the MLP's inner activations: every k_proj, v_proj, and down_proj or fc2. (PEFT's default for Qwen2 is q_proj, v_proj and down_proj; for Llama it is the paper's.)


The documentation for this struct was generated from the following file: