Adam (Kingma & Ba, 2015): per-parameter moving averages of gradient (m) and squared gradient (v), with bias correction.
More...
#include <adam_optimizer.hpp>
|
| struct | AdamState |
| | One parameter's Adam state: first and second moments, and its step count. More...
|
| |
|
| | AdamOptimizer (float learning_rate, DeviceBackend *backend, float beta1=0.9f, float beta2=0.999f, float eps=1e-8f) |
| | Constructs an Adam optimizer.
|
| |
| void | step (Module &module) |
| | Applies one Adam update to every parameter the module exposes.
|
| |
| void | zero_grad (Module &module) |
| | Resets every parameter's gradient to zero. Does not reset Adam's moment state.
|
| |
| float | learning_rate () const |
| | Current step size.
|
| |
| void | set_learning_rate (float learning_rate) |
| | Overwrites the step size used by every subsequent step() call – necessary infrastructure for any mid-training hyperparameter schedule (e.g. Population Based Training's own explore step), found necessary by campaign_exai_dl_library_evolutionary_deep_learning's Phase 4 Mission 0, logged as a small addition beyond this class's original fixed-at-construction scope. Does not reset Adam's own moment state (m/v), matching zero_grad()'s own precedent that state and gradient are independent concerns.
|
| |
| float | weight_decay () const |
| | The default group's weight decay: L2 added to the gradient, as PyTorch's Adam does (decoupled decay is AdamW, TRN-2). Defaults to 0.
|
| |
| void | set_weight_decay (float weight_decay) |
| |
| void | set_param_groups (std::vector< ParamGroup > groups) |
| | Replaces the parameter groups; see ParamGroup.
|
| |
| std::vector< ParamGroup > & | param_groups () |
| |
| const AdamState * | state (const Tensor *parameter) const |
| | The state for parameter (a value tensor from Module::parameters()), or nullptr if step() hasn't updated it yet. For checkpointing (IO-2).
|
| |
| void | set_state (const Tensor *parameter, AdamState state) |
| | Replaces the state for parameter, e.g. when resuming from a checkpoint.
|
| |
Adam (Kingma & Ba, 2015): per-parameter moving averages of gradient (m) and squared gradient (v), with bias correction.
- Note
- Per-parameter state is keyed by the parameter Tensor's pointer identity (
ParamRef::value), which is stable for as long as the owning Module exists – LinearModule/Conv2DModule's weight_/bias_ members never move or get reallocated.
◆ AdamOptimizer()
| pulsatrix::AdamOptimizer::AdamOptimizer |
( |
float |
learning_rate, |
|
|
DeviceBackend * |
backend, |
|
|
float |
beta1 = 0.9f, |
|
|
float |
beta2 = 0.999f, |
|
|
float |
eps = 1e-8f |
|
) |
| |
|
explicit |
Constructs an Adam optimizer.
- Parameters
-
| learning_rate | Step size. |
| backend | Backend used to allocate per-parameter moment-tracking tensors. |
| beta1 | First moment decay rate. |
| beta2 | Second moment decay rate. |
| eps | Denominator stabilizer. |
◆ learning_rate()
| float pulsatrix::AdamOptimizer::learning_rate |
( |
| ) |
const |
|
inline |
◆ param_groups()
| std::vector< ParamGroup > & pulsatrix::AdamOptimizer::param_groups |
( |
| ) |
|
|
inline |
◆ set_decoupled_weight_decay()
| void pulsatrix::AdamOptimizer::set_decoupled_weight_decay |
( |
bool |
decoupled | ) |
|
|
inlineprotected |
Switches weight decay from L2 (Adam) to decoupled (AdamW).
◆ set_learning_rate()
| void pulsatrix::AdamOptimizer::set_learning_rate |
( |
float |
learning_rate | ) |
|
|
inline |
Overwrites the step size used by every subsequent step() call – necessary infrastructure for any mid-training hyperparameter schedule (e.g. Population Based Training's own explore step), found necessary by campaign_exai_dl_library_evolutionary_deep_learning's Phase 4 Mission 0, logged as a small addition beyond this class's original fixed-at-construction scope. Does not reset Adam's own moment state (m/v), matching zero_grad()'s own precedent that state and gradient are independent concerns.
◆ set_param_groups()
| void pulsatrix::AdamOptimizer::set_param_groups |
( |
std::vector< ParamGroup > |
groups | ) |
|
|
inline |
Replaces the parameter groups; see ParamGroup.
- Exceptions
-
| std::invalid_argument | on a group without a selector or with an invalid setting. |
◆ set_state()
| void pulsatrix::AdamOptimizer::set_state |
( |
const Tensor * |
parameter, |
|
|
AdamState |
state |
|
) |
| |
|
inline |
Replaces the state for parameter, e.g. when resuming from a checkpoint.
- Exceptions
-
| std::invalid_argument | if the moments' shapes differ from the parameter's, they are not on the parameter's device, or t is negative. |
◆ set_weight_decay()
| void pulsatrix::AdamOptimizer::set_weight_decay |
( |
float |
weight_decay | ) |
|
|
inline |
◆ state()
| const AdamState * pulsatrix::AdamOptimizer::state |
( |
const Tensor * |
parameter | ) |
const |
|
inline |
The state for parameter (a value tensor from Module::parameters()), or nullptr if step() hasn't updated it yet. For checkpointing (IO-2).
◆ step()
| void pulsatrix::AdamOptimizer::step |
( |
Module & |
module | ) |
|
Applies one Adam update to every parameter the module exposes.
- Parameters
-
| module | Module to update. Safe no-op if it has no parameters. |
- Note
- Device-generic: one fused DeviceBackend::adam_step per parameter. Moments are allocated through this optimizer's backend, so every parameter must live on that backend's device.
- Exceptions
-
| std::invalid_argument | if a parameter's device differs from the backend's. |
◆ weight_decay()
| float pulsatrix::AdamOptimizer::weight_decay |
( |
| ) |
const |
|
inline |
The default group's weight decay: L2 added to the gradient, as PyTorch's Adam does (decoupled decay is AdamW, TRN-2). Defaults to 0.
- Exceptions
-
| std::invalid_argument | if negative or not finite. |
◆ zero_grad()
| void pulsatrix::AdamOptimizer::zero_grad |
( |
Module & |
module | ) |
|
Resets every parameter's gradient to zero. Does not reset Adam's moment state.
- Parameters
-
| module | Module whose gradients to reset. Safe no-op if it has no parameters. |
The documentation for this class was generated from the following file: