pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::SGDOptimizer Class Reference

param -= learning_rate * (grad + weight_decay * param), per parameter, for every parameter a Module exposes. Learning rate and weight decay can differ per parameter group (TRN-1). With no weight decay this is plain SGD. More...

#include <sgd_optimizer.hpp>

Public Member Functions

 SGDOptimizer (float learning_rate, float momentum=0.0f, bool nesterov=false)
 Constructs an SGD optimizer.
 
void step (Module &module)
 Applies one SGD update to every parameter the module exposes.
 
void zero_grad (Module &module)
 Resets every parameter's gradient to zero.
 
float learning_rate () const
 The default group's learning rate (parameters no group selects).
 
void set_learning_rate (float learning_rate)
 
float weight_decay () const
 The default group's weight decay. Defaults to 0.
 
void set_weight_decay (float weight_decay)
 
void set_param_groups (std::vector< ParamGroup > groups)
 Replaces the parameter groups; see ParamGroup.
 
std::vector< ParamGroup > & param_groups ()
 
float momentum () const
 
bool nesterov () const
 
const Tensor * momentum_buffer (const Tensor *parameter) const
 The momentum buffer for parameter, or nullptr before its first step (or with no momentum).
 

Detailed Description

param -= learning_rate * (grad + weight_decay * param), per parameter, for every parameter a Module exposes. Learning rate and weight decay can differ per parameter group (TRN-1). With no weight decay this is plain SGD.

Note
Optional momentum and Nesterov momentum follow torch.optim.SGD (dampening 0): with d = grad + weight_decay * param, the buffer is buf = d on a parameter's first step and buf = momentum * buf + d after; the step is -lr * buf, or -lr * (d + momentum * buf) with Nesterov (TRN-2).

Constructor & Destructor Documentation

◆ SGDOptimizer()

pulsatrix::SGDOptimizer::SGDOptimizer ( float  learning_rate,
float  momentum = 0.0f,
bool  nesterov = false 
)
inlineexplicit

Constructs an SGD optimizer.

Parameters
learning_rateStep size.
momentumIn [0, 1). 0 (the default) is plain SGD, with no per-parameter state.
nesterovUse Nesterov momentum. Needs momentum > 0.
Exceptions
std::invalid_argumentif momentum is outside [0, 1), or nesterov without momentum.

Member Function Documentation

◆ learning_rate()

float pulsatrix::SGDOptimizer::learning_rate ( ) const
inline

The default group's learning rate (parameters no group selects).

◆ momentum()

float pulsatrix::SGDOptimizer::momentum ( ) const
inline

◆ momentum_buffer()

const Tensor * pulsatrix::SGDOptimizer::momentum_buffer ( const Tensor *  parameter) const
inline

The momentum buffer for parameter, or nullptr before its first step (or with no momentum).

◆ nesterov()

bool pulsatrix::SGDOptimizer::nesterov ( ) const
inline

◆ param_groups()

std::vector< ParamGroup > & pulsatrix::SGDOptimizer::param_groups ( )
inline

◆ set_learning_rate()

void pulsatrix::SGDOptimizer::set_learning_rate ( float  learning_rate)
inline

◆ set_param_groups()

void pulsatrix::SGDOptimizer::set_param_groups ( std::vector< ParamGroup >  groups)
inline

Replaces the parameter groups; see ParamGroup.

Exceptions
std::invalid_argumenton a group without a selector or with an invalid setting.

◆ set_weight_decay()

void pulsatrix::SGDOptimizer::set_weight_decay ( float  weight_decay)
inline

◆ step()

void pulsatrix::SGDOptimizer::step ( Module &  module)

Applies one SGD update to every parameter the module exposes.

Parameters
moduleModule to update. Safe no-op if it has no parameters.
Note
Device-generic: runs on Cpu, Cuda or Hip tensors (GPU-native-kernels Mission 1).

◆ weight_decay()

float pulsatrix::SGDOptimizer::weight_decay ( ) const
inline

The default group's weight decay. Defaults to 0.

Exceptions
std::invalid_argumentif negative or not finite.

◆ zero_grad()

void pulsatrix::SGDOptimizer::zero_grad ( Module &  module)

Resets every parameter's gradient to zero.

Parameters
moduleModule whose gradients to reset. Safe no-op if it has no parameters.

The documentation for this class was generated from the following file: