A learning-rate multiplier as a function of the step, with the formulas of Hugging Face transformers' get_{constant,linear,cosine}_schedule_with_warmup.
More...
#include <lr_scheduler.hpp>
|
| float | multiplier (int64_t step) const |
| | The multiplier at step.
|
| |
|
| static LRSchedule | Constant (int64_t warmup=0) |
| | 1 after an optional linear warmup.
|
| |
| static LRSchedule | Linear (int64_t warmup, int64_t total) |
| | Warmup, then linear decay to 0 at total.
|
| |
| static LRSchedule | Cosine (int64_t warmup, int64_t total, float min_ratio=0.0f) |
| | Warmup, then cosine decay to min_ratio at total.
|
| |
A learning-rate multiplier as a function of the step, with the formulas of Hugging Face transformers' get_{constant,linear,cosine}_schedule_with_warmup.
- Note
- Warmup ramps linearly from 0 at step 0 to 1 at step
warmup. Linear then decays to 0 at step total; cosine follows half a cosine down to min_ratio at total. Both stay at their final value past total.
◆ Constant()
| static LRSchedule pulsatrix::LRSchedule::Constant |
( |
int64_t |
warmup = 0 | ) |
|
|
inlinestatic |
1 after an optional linear warmup.
- Exceptions
-
| std::invalid_argument | if warmup < 0. |
◆ Cosine()
| static LRSchedule pulsatrix::LRSchedule::Cosine |
( |
int64_t |
warmup, |
|
|
int64_t |
total, |
|
|
float |
min_ratio = 0.0f |
|
) |
| |
|
inlinestatic |
Warmup, then cosine decay to min_ratio at total.
- Exceptions
-
| std::invalid_argument | unless 0 <= warmup <= total, total > 0 and min_ratio is in [0, 1]. |
◆ Linear()
| static LRSchedule pulsatrix::LRSchedule::Linear |
( |
int64_t |
warmup, |
|
|
int64_t |
total |
|
) |
| |
|
inlinestatic |
Warmup, then linear decay to 0 at total.
- Exceptions
-
| std::invalid_argument | unless 0 <= warmup <= total and total > 0. |
◆ multiplier()
| float pulsatrix::LRSchedule::multiplier |
( |
int64_t |
step | ) |
const |
|
inline |
The multiplier at step.
- Exceptions
-
| std::invalid_argument | if step < 0. |
The documentation for this class was generated from the following file: