8#include <unordered_map>
91 auto it = state_.find(parameter);
92 return it == state_.end() ? nullptr : &it->second;
102 throw std::invalid_argument(
"AdamOptimizer::set_state: moments must have the parameter's shape");
105 throw std::invalid_argument(
"AdamOptimizer::set_state: moments must be on the parameter's device");
108 throw std::invalid_argument(
"AdamOptimizer::set_state: step count must be non-negative");
110 state_.insert_or_assign(parameter, std::move(
state));
118 float learning_rate_;
119 float weight_decay_ = 0.0f;
120 bool decoupled_weight_decay_ =
false;
126 std::unordered_map<const Tensor*, AdamState> state_;
144 float beta2 = 0.999f,
float eps = 1e-8f)
Adam (Kingma & Ba, 2015): per-parameter moving averages of gradient (m) and squared gradient (v),...
Definition adam_optimizer.hpp:22
void set_weight_decay(float weight_decay)
Definition adam_optimizer.hpp:69
void set_state(const Tensor *parameter, AdamState state)
Replaces the state for parameter, e.g. when resuming from a checkpoint.
Definition adam_optimizer.hpp:100
void step(Module &module)
Applies one Adam update to every parameter the module exposes.
void set_param_groups(std::vector< ParamGroup > groups)
Replaces the parameter groups; see ParamGroup.
Definition adam_optimizer.hpp:76
void set_decoupled_weight_decay(bool decoupled)
Switches weight decay from L2 (Adam) to decoupled (AdamW).
Definition adam_optimizer.hpp:115
std::vector< ParamGroup > & param_groups()
Definition adam_optimizer.hpp:77
void zero_grad(Module &module)
Resets every parameter's gradient to zero. Does not reset Adam's moment state.
void set_learning_rate(float learning_rate)
Overwrites the step size used by every subsequent step() call – necessary infrastructure for any mid-...
Definition adam_optimizer.hpp:63
AdamOptimizer(float learning_rate, DeviceBackend *backend, float beta1=0.9f, float beta2=0.999f, float eps=1e-8f)
Constructs an Adam optimizer.
const AdamState * state(const Tensor *parameter) const
The state for parameter (a value tensor from Module::parameters()), or nullptr if step() hasn't updat...
Definition adam_optimizer.hpp:90
float weight_decay() const
The default group's weight decay: L2 added to the gradient, as PyTorch's Adam does (decoupled decay i...
Definition adam_optimizer.hpp:68
float learning_rate() const
Current step size.
Definition adam_optimizer.hpp:52
AdamW (Loshchilov & Hutter, "Decoupled Weight Decay Regularization", arXiv 1711.05101): Adam whose we...
Definition adam_optimizer.hpp:137
AdamWOptimizer(float learning_rate, DeviceBackend *backend, float weight_decay=0.01f, float beta1=0.9f, float beta2=0.999f, float eps=1e-8f)
Definition adam_optimizer.hpp:143
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
Base class for every layer type (LinearModule, Conv2DModule, activations, ...).
Definition module.hpp:78
The group machinery SGDOptimizer and AdamOptimizer share.
Definition param_groups.hpp:67
std::vector< ParamGroup > & groups()
Definition param_groups.hpp:84
void set(std::vector< ParamGroup > groups)
Definition param_groups.hpp:77
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
DeviceType device() const
Which device this tensor's buffer conceptually resides on.
Definition tensor.hpp:122
const Shape & shape() const
This tensor's shape.
Definition tensor.hpp:113
Abstract base every layer subclasses – NVI forward(), pure-virtual LRP contract.
Definition acquisition_functions.hpp:16
void check_optimizer_setting(float value, const std::string &what)
Throws std::invalid_argument unless value is finite and non-negative.
Definition param_groups.hpp:60
Optimizer parameter groups: a learning rate and weight decay per set of parameters,...
One parameter's Adam state: first and second moments, and its step count.
Definition adam_optimizer.hpp:80
Tensor v
Definition adam_optimizer.hpp:82
int64_t t
Definition adam_optimizer.hpp:83
Tensor m
Definition adam_optimizer.hpp:81