param -= learning_rate * (grad + weight_decay * param), per parameter, for every parameter a Module exposes. Learning rate and weight decay can differ per parameter group (TRN-1). With no weight decay this is plain SGD.
More...
#include <sgd_optimizer.hpp>
param -= learning_rate * (grad + weight_decay * param), per parameter, for every parameter a Module exposes. Learning rate and weight decay can differ per parameter group (TRN-1). With no weight decay this is plain SGD.
- Note
- Optional momentum and Nesterov momentum follow torch.optim.SGD (dampening 0): with d = grad + weight_decay * param, the buffer is buf = d on a parameter's first step and buf = momentum * buf + d after; the step is -lr * buf, or -lr * (d + momentum * buf) with Nesterov (TRN-2).
◆ SGDOptimizer()
| pulsatrix::SGDOptimizer::SGDOptimizer |
( |
float |
learning_rate, |
|
|
float |
momentum = 0.0f, |
|
|
bool |
nesterov = false |
|
) |
| |
|
inlineexplicit |
Constructs an SGD optimizer.
- Parameters
-
| learning_rate | Step size. |
| momentum | In [0, 1). 0 (the default) is plain SGD, with no per-parameter state. |
| nesterov | Use Nesterov momentum. Needs momentum > 0. |
- Exceptions
-
| std::invalid_argument | if momentum is outside [0, 1), or nesterov without momentum. |
◆ learning_rate()
| float pulsatrix::SGDOptimizer::learning_rate |
( |
| ) |
const |
|
inline |
The default group's learning rate (parameters no group selects).
◆ momentum()
| float pulsatrix::SGDOptimizer::momentum |
( |
| ) |
const |
|
inline |
◆ momentum_buffer()
| const Tensor * pulsatrix::SGDOptimizer::momentum_buffer |
( |
const Tensor * |
parameter | ) |
const |
|
inline |
The momentum buffer for parameter, or nullptr before its first step (or with no momentum).
◆ nesterov()
| bool pulsatrix::SGDOptimizer::nesterov |
( |
| ) |
const |
|
inline |
◆ param_groups()
| std::vector< ParamGroup > & pulsatrix::SGDOptimizer::param_groups |
( |
| ) |
|
|
inline |
◆ set_learning_rate()
| void pulsatrix::SGDOptimizer::set_learning_rate |
( |
float |
learning_rate | ) |
|
|
inline |
◆ set_param_groups()
| void pulsatrix::SGDOptimizer::set_param_groups |
( |
std::vector< ParamGroup > |
groups | ) |
|
|
inline |
Replaces the parameter groups; see ParamGroup.
- Exceptions
-
| std::invalid_argument | on a group without a selector or with an invalid setting. |
◆ set_weight_decay()
| void pulsatrix::SGDOptimizer::set_weight_decay |
( |
float |
weight_decay | ) |
|
|
inline |
◆ step()
| void pulsatrix::SGDOptimizer::step |
( |
Module & |
module | ) |
|
Applies one SGD update to every parameter the module exposes.
- Parameters
-
| module | Module to update. Safe no-op if it has no parameters. |
- Note
- Device-generic: runs on Cpu, Cuda or Hip tensors (GPU-native-kernels Mission 1).
◆ weight_decay()
| float pulsatrix::SGDOptimizer::weight_decay |
( |
| ) |
const |
|
inline |
The default group's weight decay. Defaults to 0.
- Exceptions
-
| std::invalid_argument | if negative or not finite. |
◆ zero_grad()
| void pulsatrix::SGDOptimizer::zero_grad |
( |
Module & |
module | ) |
|
Resets every parameter's gradient to zero.
- Parameters
-
| module | Module whose gradients to reset. Safe no-op if it has no parameters. |
The documentation for this class was generated from the following file: