pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::AdamWOptimizer Class Reference

AdamW (Loshchilov & Hutter, "Decoupled Weight Decay Regularization", arXiv 1711.05101): Adam whose weight decay shrinks the weights directly, w <- (1 - lr * wd) * w, before the Adam step, instead of adding wd * w to the gradient (roadmap TRN-2). More...

#include <adam_optimizer.hpp>

Inheritance diagram for pulsatrix::AdamWOptimizer:
Collaboration diagram for pulsatrix::AdamWOptimizer:

Public Member Functions

 AdamWOptimizer (float learning_rate, DeviceBackend *backend, float weight_decay=0.01f, float beta1=0.9f, float beta2=0.999f, float eps=1e-8f)
 
- Public Member Functions inherited from pulsatrix::AdamOptimizer
 AdamOptimizer (float learning_rate, DeviceBackend *backend, float beta1=0.9f, float beta2=0.999f, float eps=1e-8f)
 Constructs an Adam optimizer.
 
void step (Module &module)
 Applies one Adam update to every parameter the module exposes.
 
void zero_grad (Module &module)
 Resets every parameter's gradient to zero. Does not reset Adam's moment state.
 
float learning_rate () const
 Current step size.
 
void set_learning_rate (float learning_rate)
 Overwrites the step size used by every subsequent step() call – necessary infrastructure for any mid-training hyperparameter schedule (e.g. Population Based Training's own explore step), found necessary by campaign_exai_dl_library_evolutionary_deep_learning's Phase 4 Mission 0, logged as a small addition beyond this class's original fixed-at-construction scope. Does not reset Adam's own moment state (m/v), matching zero_grad()'s own precedent that state and gradient are independent concerns.
 
float weight_decay () const
 The default group's weight decay: L2 added to the gradient, as PyTorch's Adam does (decoupled decay is AdamW, TRN-2). Defaults to 0.
 
void set_weight_decay (float weight_decay)
 
void set_param_groups (std::vector< ParamGroup > groups)
 Replaces the parameter groups; see ParamGroup.
 
std::vector< ParamGroup > & param_groups ()
 
const AdamState * state (const Tensor *parameter) const
 The state for parameter (a value tensor from Module::parameters()), or nullptr if step() hasn't updated it yet. For checkpointing (IO-2).
 
void set_state (const Tensor *parameter, AdamState state)
 Replaces the state for parameter, e.g. when resuming from a checkpoint.
 

Additional Inherited Members

- Protected Member Functions inherited from pulsatrix::AdamOptimizer
void set_decoupled_weight_decay (bool decoupled)
 Switches weight decay from L2 (Adam) to decoupled (AdamW).
 

Detailed Description

AdamW (Loshchilov & Hutter, "Decoupled Weight Decay Regularization", arXiv 1711.05101): Adam whose weight decay shrinks the weights directly, w <- (1 - lr * wd) * w, before the Adam step, instead of adding wd * w to the gradient (roadmap TRN-2).

Note
The default for transformer fine-tuning. With weight_decay = 0 it is exactly Adam. Its state is Adam's, so state(), set_state() and checkpoints (IO-2) work unchanged. Parameter groups set the decay per group, as for Adam.

Constructor & Destructor Documentation

◆ AdamWOptimizer()

pulsatrix::AdamWOptimizer::AdamWOptimizer ( float  learning_rate,
DeviceBackend *  backend,
float  weight_decay = 0.01f,
float  beta1 = 0.9f,
float  beta2 = 0.999f,
float  eps = 1e-8f 
)
inline
Parameters
weight_decayDefault group's decoupled decay. Defaults to 0.01, like torch.optim.AdamW.
Exceptions
std::invalid_argumentif weight_decay is negative or not finite.

The documentation for this class was generated from the following file: