One layer's low-rank update: A (in, r) and B (r, out), stored like LinearModule's weight, with x A B as the update. PEFT's lora_A and lora_B are the transposes, (r, in) and (out, r); SaveLoraAdapter() and LoadLoraAdapter() convert.
More...
#include <lora.hpp>
|
| | LoRAAdapter (int64_t in_features, int64_t out_features, const LoRAConfig &config, DeviceBackend *backend, DeviceType device) |
| | A as PEFT initializes it (Kaiming-uniform with a = sqrt(5), so uniform on ±1 / sqrt(in)), and B zero.
|
| |
| int64_t | rank () const |
| |
| float | alpha () const |
| |
| float | scale () const |
| | alpha / rank.
|
| |
| float | dropout () const |
| |
| const Tensor & | a () const |
| |
| const Tensor & | b () const |
| |
| const Tensor & | a_grad () const |
| |
| const Tensor & | b_grad () const |
| |
| void | set_a (const std::vector< float > &values) |
| | Replaces A, (in, r) row-major.
|
| |
| void | set_b (const std::vector< float > &values) |
| | Replaces B, (r, out) row-major.
|
| |
| bool | merged () const |
| | True while the update is added into the base weight (LinearModule::merge_lora()).
|
| |
| bool | enabled () const |
| | False while the layer runs as its base layer (LinearModule::set_lora_enabled()).
|
| |
| Tensor | effective_weight (const Tensor &weight) const |
| | The adapted weight W + scale() A B, as a new tensor.
|
| |
| Tensor | update () const |
| | The update scale() A B alone, (in, out).
|
| |
| bool | has_relevance () const |
| | True after a relevance pass that split this layer (LoRARelevance::Split, adapter enabled and unmerged), until the next forward().
|
| |
| const Tensor & | last_base_relevance () const |
| | The last split's input relevance through the base weight W, (N, in).
|
| |
| const Tensor & | last_adapter_relevance () const |
| | The last split's input relevance through the adapter, (N, in): what the fine-tuning contributes to this layer's explanation. The two parts sum to the layer's input relevance.
|
| |
One layer's low-rank update: A (in, r) and B (r, out), stored like LinearModule's weight, with x A B as the update. PEFT's lora_A and lora_B are the transposes, (r, in) and (out, r); SaveLoraAdapter() and LoadLoraAdapter() convert.
- Note
- LinearModule owns and drives it; use LinearModule::lora() to read it.
◆ LoRAAdapter()
A as PEFT initializes it (Kaiming-uniform with a = sqrt(5), so uniform on ±1 / sqrt(in)), and B zero.
- Exceptions
-
| std::invalid_argument | if rank < 1, alpha isn't finite, or dropout isn't in [0, 1). |
◆ a()
| const Tensor & pulsatrix::LoRAAdapter::a |
( |
| ) |
const |
|
inline |
◆ a_grad()
| const Tensor & pulsatrix::LoRAAdapter::a_grad |
( |
| ) |
const |
|
inline |
◆ alpha()
| float pulsatrix::LoRAAdapter::alpha |
( |
| ) |
const |
|
inline |
◆ b()
| const Tensor & pulsatrix::LoRAAdapter::b |
( |
| ) |
const |
|
inline |
◆ b_grad()
| const Tensor & pulsatrix::LoRAAdapter::b_grad |
( |
| ) |
const |
|
inline |
◆ dropout()
| float pulsatrix::LoRAAdapter::dropout |
( |
| ) |
const |
|
inline |
◆ effective_weight()
| Tensor pulsatrix::LoRAAdapter::effective_weight |
( |
const Tensor & |
weight | ) |
const |
The adapted weight W + scale() A B, as a new tensor.
◆ enabled()
| bool pulsatrix::LoRAAdapter::enabled |
( |
| ) |
const |
|
inline |
◆ has_relevance()
| bool pulsatrix::LoRAAdapter::has_relevance |
( |
| ) |
const |
|
inline |
True after a relevance pass that split this layer (LoRARelevance::Split, adapter enabled and unmerged), until the next forward().
◆ last_adapter_relevance()
| const Tensor & pulsatrix::LoRAAdapter::last_adapter_relevance |
( |
| ) |
const |
The last split's input relevance through the adapter, (N, in): what the fine-tuning contributes to this layer's explanation. The two parts sum to the layer's input relevance.
- Exceptions
-
◆ last_base_relevance()
| const Tensor & pulsatrix::LoRAAdapter::last_base_relevance |
( |
| ) |
const |
The last split's input relevance through the base weight W, (N, in).
- Exceptions
-
◆ merged()
| bool pulsatrix::LoRAAdapter::merged |
( |
| ) |
const |
|
inline |
◆ rank()
| int64_t pulsatrix::LoRAAdapter::rank |
( |
| ) |
const |
|
inline |
◆ scale()
| float pulsatrix::LoRAAdapter::scale |
( |
| ) |
const |
|
inline |
◆ set_a()
| void pulsatrix::LoRAAdapter::set_a |
( |
const std::vector< float > & |
values | ) |
|
Replaces A, (in, r) row-major.
- Exceptions
-
| std::invalid_argument | on a wrong size. |
◆ set_b()
| void pulsatrix::LoRAAdapter::set_b |
( |
const std::vector< float > & |
values | ) |
|
Replaces B, (r, out) row-major.
- Exceptions
-
| std::invalid_argument | on a wrong size. |
◆ update()
| Tensor pulsatrix::LoRAAdapter::update |
( |
| ) |
const |
The update scale() A B alone, (in, out).
◆ LinearModule
The documentation for this class was generated from the following file: