AdamW (Loshchilov & Hutter, "Decoupled Weight Decay Regularization", arXiv 1711.05101): Adam whose weight decay shrinks the weights directly, w <- (1 - lr * wd) * w, before the Adam step, instead of adding wd * w to the gradient (roadmap TRN-2).
More...
#include <adam_optimizer.hpp>
|
| | AdamWOptimizer (float learning_rate, DeviceBackend *backend, float weight_decay=0.01f, float beta1=0.9f, float beta2=0.999f, float eps=1e-8f) |
| |
| | AdamOptimizer (float learning_rate, DeviceBackend *backend, float beta1=0.9f, float beta2=0.999f, float eps=1e-8f) |
| | Constructs an Adam optimizer.
|
| |
| void | step (Module &module) |
| | Applies one Adam update to every parameter the module exposes.
|
| |
| void | zero_grad (Module &module) |
| | Resets every parameter's gradient to zero. Does not reset Adam's moment state.
|
| |
| float | learning_rate () const |
| | Current step size.
|
| |
| void | set_learning_rate (float learning_rate) |
| | Overwrites the step size used by every subsequent step() call – necessary infrastructure for any mid-training hyperparameter schedule (e.g. Population Based Training's own explore step), found necessary by campaign_exai_dl_library_evolutionary_deep_learning's Phase 4 Mission 0, logged as a small addition beyond this class's original fixed-at-construction scope. Does not reset Adam's own moment state (m/v), matching zero_grad()'s own precedent that state and gradient are independent concerns.
|
| |
| float | weight_decay () const |
| | The default group's weight decay: L2 added to the gradient, as PyTorch's Adam does (decoupled decay is AdamW, TRN-2). Defaults to 0.
|
| |
| void | set_weight_decay (float weight_decay) |
| |
| void | set_param_groups (std::vector< ParamGroup > groups) |
| | Replaces the parameter groups; see ParamGroup.
|
| |
| std::vector< ParamGroup > & | param_groups () |
| |
| const AdamState * | state (const Tensor *parameter) const |
| | The state for parameter (a value tensor from Module::parameters()), or nullptr if step() hasn't updated it yet. For checkpointing (IO-2).
|
| |
| void | set_state (const Tensor *parameter, AdamState state) |
| | Replaces the state for parameter, e.g. when resuming from a checkpoint.
|
| |
AdamW (Loshchilov & Hutter, "Decoupled Weight Decay Regularization", arXiv 1711.05101): Adam whose weight decay shrinks the weights directly, w <- (1 - lr * wd) * w, before the Adam step, instead of adding wd * w to the gradient (roadmap TRN-2).
- Note
- The default for transformer fine-tuning. With weight_decay = 0 it is exactly Adam. Its state is Adam's, so state(), set_state() and checkpoints (IO-2) work unchanged. Parameter groups set the decay per group, as for Adam.
◆ AdamWOptimizer()
| pulsatrix::AdamWOptimizer::AdamWOptimizer |
( |
float |
learning_rate, |
|
|
DeviceBackend * |
backend, |
|
|
float |
weight_decay = 0.01f, |
|
|
float |
beta1 = 0.9f, |
|
|
float |
beta2 = 0.999f, |
|
|
float |
eps = 1e-8f |
|
) |
| |
|
inline |
- Parameters
-
| weight_decay | Default group's decoupled decay. Defaults to 0.01, like torch.optim.AdamW. |
- Exceptions
-
| std::invalid_argument | if weight_decay is negative or not finite. |
The documentation for this class was generated from the following file: