pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::LRSchedule Class Reference

A learning-rate multiplier as a function of the step, with the formulas of Hugging Face transformers' get_{constant,linear,cosine}_schedule_with_warmup. More...

#include <lr_scheduler.hpp>

Public Member Functions

float multiplier (int64_t step) const
 The multiplier at step.
 

Static Public Member Functions

static LRSchedule Constant (int64_t warmup=0)
 1 after an optional linear warmup.
 
static LRSchedule Linear (int64_t warmup, int64_t total)
 Warmup, then linear decay to 0 at total.
 
static LRSchedule Cosine (int64_t warmup, int64_t total, float min_ratio=0.0f)
 Warmup, then cosine decay to min_ratio at total.
 

Detailed Description

A learning-rate multiplier as a function of the step, with the formulas of Hugging Face transformers' get_{constant,linear,cosine}_schedule_with_warmup.

Note
Warmup ramps linearly from 0 at step 0 to 1 at step warmup. Linear then decays to 0 at step total; cosine follows half a cosine down to min_ratio at total. Both stay at their final value past total.

Member Function Documentation

◆ Constant()

static LRSchedule pulsatrix::LRSchedule::Constant ( int64_t  warmup = 0)
inlinestatic

1 after an optional linear warmup.

Exceptions
std::invalid_argumentif warmup < 0.

◆ Cosine()

static LRSchedule pulsatrix::LRSchedule::Cosine ( int64_t  warmup,
int64_t  total,
float  min_ratio = 0.0f 
)
inlinestatic

Warmup, then cosine decay to min_ratio at total.

Exceptions
std::invalid_argumentunless 0 <= warmup <= total, total > 0 and min_ratio is in [0, 1].

◆ Linear()

static LRSchedule pulsatrix::LRSchedule::Linear ( int64_t  warmup,
int64_t  total 
)
inlinestatic

Warmup, then linear decay to 0 at total.

Exceptions
std::invalid_argumentunless 0 <= warmup <= total and total > 0.

◆ multiplier()

float pulsatrix::LRSchedule::multiplier ( int64_t  step) const
inline

The multiplier at step.

Exceptions
std::invalid_argumentif step < 0.

The documentation for this class was generated from the following file: