|
pulsatrix 1.1.0
|
Which layers IA³ rescales, PEFT style (see LoRAConfig::target_modules for the matching). More...
#include <ia3.hpp>
Public Attributes | |
| std::vector< std::string > | target_modules |
The layers to rescale. Empty: the paper's choice, keys, values and the MLP's inner activations: every k_proj, v_proj, and down_proj or fc2. (PEFT's default for Qwen2 is q_proj, v_proj and down_proj; for Llama it is the paper's.) | |
| std::vector< std::string > | feedforward_modules |
Those whose input is rescaled rather than their output. Each must be among the targets. Empty with empty targets: down_proj and fc2. | |
Which layers IA³ rescales, PEFT style (see LoRAConfig::target_modules for the matching).
| std::vector<std::string> pulsatrix::IA3Config::feedforward_modules |
Those whose input is rescaled rather than their output. Each must be among the targets. Empty with empty targets: down_proj and fc2.
| std::vector<std::string> pulsatrix::IA3Config::target_modules |
The layers to rescale. Empty: the paper's choice, keys, values and the MLP's inner activations: every k_proj, v_proj, and down_proj or fc2. (PEFT's default for Qwen2 is q_proj, v_proj and down_proj; for Llama it is the paper's.)