pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::LoRAAdapter Class Reference

One layer's low-rank update: A (in, r) and B (r, out), stored like LinearModule's weight, with x A B as the update. PEFT's lora_A and lora_B are the transposes, (r, in) and (out, r); SaveLoraAdapter() and LoadLoraAdapter() convert. More...

#include <lora.hpp>

Public Member Functions

 LoRAAdapter (int64_t in_features, int64_t out_features, const LoRAConfig &config, DeviceBackend *backend, DeviceType device)
 A as PEFT initializes it (Kaiming-uniform with a = sqrt(5), so uniform on ±1 / sqrt(in)), and B zero.
 
int64_t rank () const
 
float alpha () const
 
float scale () const
 alpha / rank.
 
float dropout () const
 
const Tensor & a () const
 
const Tensor & b () const
 
const Tensor & a_grad () const
 
const Tensor & b_grad () const
 
void set_a (const std::vector< float > &values)
 Replaces A, (in, r) row-major.
 
void set_b (const std::vector< float > &values)
 Replaces B, (r, out) row-major.
 
bool merged () const
 True while the update is added into the base weight (LinearModule::merge_lora()).
 
bool enabled () const
 False while the layer runs as its base layer (LinearModule::set_lora_enabled()).
 
Tensor effective_weight (const Tensor &weight) const
 The adapted weight W + scale() A B, as a new tensor.
 
Tensor update () const
 The update scale() A B alone, (in, out).
 
bool has_relevance () const
 True after a relevance pass that split this layer (LoRARelevance::Split, adapter enabled and unmerged), until the next forward().
 
const Tensor & last_base_relevance () const
 The last split's input relevance through the base weight W, (N, in).
 
const Tensor & last_adapter_relevance () const
 The last split's input relevance through the adapter, (N, in): what the fine-tuning contributes to this layer's explanation. The two parts sum to the layer's input relevance.
 

Friends

class LinearModule
 

Detailed Description

One layer's low-rank update: A (in, r) and B (r, out), stored like LinearModule's weight, with x A B as the update. PEFT's lora_A and lora_B are the transposes, (r, in) and (out, r); SaveLoraAdapter() and LoadLoraAdapter() convert.

Note
LinearModule owns and drives it; use LinearModule::lora() to read it.

Constructor & Destructor Documentation

◆ LoRAAdapter()

pulsatrix::LoRAAdapter::LoRAAdapter ( int64_t  in_features,
int64_t  out_features,
const LoRAConfig &  config,
DeviceBackend *  backend,
DeviceType  device 
)

A as PEFT initializes it (Kaiming-uniform with a = sqrt(5), so uniform on ±1 / sqrt(in)), and B zero.

Exceptions
std::invalid_argumentif rank < 1, alpha isn't finite, or dropout isn't in [0, 1).

Member Function Documentation

◆ a()

const Tensor & pulsatrix::LoRAAdapter::a ( ) const
inline

◆ a_grad()

const Tensor & pulsatrix::LoRAAdapter::a_grad ( ) const
inline

◆ alpha()

float pulsatrix::LoRAAdapter::alpha ( ) const
inline

◆ b()

const Tensor & pulsatrix::LoRAAdapter::b ( ) const
inline

◆ b_grad()

const Tensor & pulsatrix::LoRAAdapter::b_grad ( ) const
inline

◆ dropout()

float pulsatrix::LoRAAdapter::dropout ( ) const
inline

◆ effective_weight()

Tensor pulsatrix::LoRAAdapter::effective_weight ( const Tensor &  weight) const

The adapted weight W + scale() A B, as a new tensor.

◆ enabled()

bool pulsatrix::LoRAAdapter::enabled ( ) const
inline

False while the layer runs as its base layer (LinearModule::set_lora_enabled()).

◆ has_relevance()

bool pulsatrix::LoRAAdapter::has_relevance ( ) const
inline

True after a relevance pass that split this layer (LoRARelevance::Split, adapter enabled and unmerged), until the next forward().

◆ last_adapter_relevance()

const Tensor & pulsatrix::LoRAAdapter::last_adapter_relevance ( ) const

The last split's input relevance through the adapter, (N, in): what the fine-tuning contributes to this layer's explanation. The two parts sum to the layer's input relevance.

Exceptions
std::logic_errorunless has_relevance().

◆ last_base_relevance()

const Tensor & pulsatrix::LoRAAdapter::last_base_relevance ( ) const

The last split's input relevance through the base weight W, (N, in).

Exceptions
std::logic_errorunless has_relevance().

◆ merged()

bool pulsatrix::LoRAAdapter::merged ( ) const
inline

True while the update is added into the base weight (LinearModule::merge_lora()).

◆ rank()

int64_t pulsatrix::LoRAAdapter::rank ( ) const
inline

◆ scale()

float pulsatrix::LoRAAdapter::scale ( ) const
inline

alpha / rank.

◆ set_a()

void pulsatrix::LoRAAdapter::set_a ( const std::vector< float > &  values)

Replaces A, (in, r) row-major.

Exceptions
std::invalid_argumenton a wrong size.

◆ set_b()

void pulsatrix::LoRAAdapter::set_b ( const std::vector< float > &  values)

Replaces B, (r, out) row-major.

Exceptions
std::invalid_argumenton a wrong size.

◆ update()

Tensor pulsatrix::LoRAAdapter::update ( ) const

The update scale() A B alone, (in, out).

Friends And Related Symbol Documentation

◆ LinearModule

friend class LinearModule
friend

The documentation for this class was generated from the following file: