pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::AdamOptimizer Class Reference

Adam (Kingma & Ba, 2015): per-parameter moving averages of gradient (m) and squared gradient (v), with bias correction. More...

#include <adam_optimizer.hpp>

Inheritance diagram for pulsatrix::AdamOptimizer:

Classes

struct  AdamState
 One parameter's Adam state: first and second moments, and its step count. More...
 

Public Member Functions

 AdamOptimizer (float learning_rate, DeviceBackend *backend, float beta1=0.9f, float beta2=0.999f, float eps=1e-8f)
 Constructs an Adam optimizer.
 
void step (Module &module)
 Applies one Adam update to every parameter the module exposes.
 
void zero_grad (Module &module)
 Resets every parameter's gradient to zero. Does not reset Adam's moment state.
 
float learning_rate () const
 Current step size.
 
void set_learning_rate (float learning_rate)
 Overwrites the step size used by every subsequent step() call – necessary infrastructure for any mid-training hyperparameter schedule (e.g. Population Based Training's own explore step), found necessary by campaign_exai_dl_library_evolutionary_deep_learning's Phase 4 Mission 0, logged as a small addition beyond this class's original fixed-at-construction scope. Does not reset Adam's own moment state (m/v), matching zero_grad()'s own precedent that state and gradient are independent concerns.
 
float weight_decay () const
 The default group's weight decay: L2 added to the gradient, as PyTorch's Adam does (decoupled decay is AdamW, TRN-2). Defaults to 0.
 
void set_weight_decay (float weight_decay)
 
void set_param_groups (std::vector< ParamGroup > groups)
 Replaces the parameter groups; see ParamGroup.
 
std::vector< ParamGroup > & param_groups ()
 
const AdamState * state (const Tensor *parameter) const
 The state for parameter (a value tensor from Module::parameters()), or nullptr if step() hasn't updated it yet. For checkpointing (IO-2).
 
void set_state (const Tensor *parameter, AdamState state)
 Replaces the state for parameter, e.g. when resuming from a checkpoint.
 

Protected Member Functions

void set_decoupled_weight_decay (bool decoupled)
 Switches weight decay from L2 (Adam) to decoupled (AdamW).
 

Detailed Description

Adam (Kingma & Ba, 2015): per-parameter moving averages of gradient (m) and squared gradient (v), with bias correction.

Note
Per-parameter state is keyed by the parameter Tensor's pointer identity (ParamRef::value), which is stable for as long as the owning Module exists – LinearModule/Conv2DModule's weight_/bias_ members never move or get reallocated.

Constructor & Destructor Documentation

◆ AdamOptimizer()

pulsatrix::AdamOptimizer::AdamOptimizer ( float  learning_rate,
DeviceBackend *  backend,
float  beta1 = 0.9f,
float  beta2 = 0.999f,
float  eps = 1e-8f 
)
explicit

Constructs an Adam optimizer.

Parameters
learning_rateStep size.
backendBackend used to allocate per-parameter moment-tracking tensors.
beta1First moment decay rate.
beta2Second moment decay rate.
epsDenominator stabilizer.

Member Function Documentation

◆ learning_rate()

float pulsatrix::AdamOptimizer::learning_rate ( ) const
inline

Current step size.

◆ param_groups()

std::vector< ParamGroup > & pulsatrix::AdamOptimizer::param_groups ( )
inline

◆ set_decoupled_weight_decay()

void pulsatrix::AdamOptimizer::set_decoupled_weight_decay ( bool  decoupled)
inlineprotected

Switches weight decay from L2 (Adam) to decoupled (AdamW).

◆ set_learning_rate()

void pulsatrix::AdamOptimizer::set_learning_rate ( float  learning_rate)
inline

Overwrites the step size used by every subsequent step() call – necessary infrastructure for any mid-training hyperparameter schedule (e.g. Population Based Training's own explore step), found necessary by campaign_exai_dl_library_evolutionary_deep_learning's Phase 4 Mission 0, logged as a small addition beyond this class's original fixed-at-construction scope. Does not reset Adam's own moment state (m/v), matching zero_grad()'s own precedent that state and gradient are independent concerns.

◆ set_param_groups()

void pulsatrix::AdamOptimizer::set_param_groups ( std::vector< ParamGroup >  groups)
inline

Replaces the parameter groups; see ParamGroup.

Exceptions
std::invalid_argumenton a group without a selector or with an invalid setting.

◆ set_state()

void pulsatrix::AdamOptimizer::set_state ( const Tensor *  parameter,
AdamState  state 
)
inline

Replaces the state for parameter, e.g. when resuming from a checkpoint.

Exceptions
std::invalid_argumentif the moments' shapes differ from the parameter's, they are not on the parameter's device, or t is negative.

◆ set_weight_decay()

void pulsatrix::AdamOptimizer::set_weight_decay ( float  weight_decay)
inline

◆ state()

const AdamState * pulsatrix::AdamOptimizer::state ( const Tensor *  parameter) const
inline

The state for parameter (a value tensor from Module::parameters()), or nullptr if step() hasn't updated it yet. For checkpointing (IO-2).

◆ step()

void pulsatrix::AdamOptimizer::step ( Module &  module)

Applies one Adam update to every parameter the module exposes.

Parameters
moduleModule to update. Safe no-op if it has no parameters.
Note
Device-generic: one fused DeviceBackend::adam_step per parameter. Moments are allocated through this optimizer's backend, so every parameter must live on that backend's device.
Exceptions
std::invalid_argumentif a parameter's device differs from the backend's.

◆ weight_decay()

float pulsatrix::AdamOptimizer::weight_decay ( ) const
inline

The default group's weight decay: L2 added to the gradient, as PyTorch's Adam does (decoupled decay is AdamW, TRN-2). Defaults to 0.

Exceptions
std::invalid_argumentif negative or not finite.

◆ zero_grad()

void pulsatrix::AdamOptimizer::zero_grad ( Module &  module)

Resets every parameter's gradient to zero. Does not reset Adam's moment state.

Parameters
moduleModule whose gradients to reset. Safe no-op if it has no parameters.

The documentation for this class was generated from the following file: