pulsatrix 1.1.0
Loading...
Searching...
No Matches
distillation_loss.hpp
Go to the documentation of this file.
1
23#pragma once
24
25#include <cstdint>
26
27#include "pulsatrix/tensor.hpp"
29
30namespace pulsatrix {
31
34 Forward,
35 Reverse,
36};
37
49
59public:
61
70 [[nodiscard]] float forward(const Tensor& student_logits, const Tensor& teacher_logits, float normalizer = 0.0f);
71
79 [[nodiscard]] float forward(const Tensor& student_logits, const Tensor& teacher_logits, const Tensor& targets,
80 float normalizer = 0.0f);
81
84 [[nodiscard]] Tensor backward() const;
85
88 [[nodiscard]] float soft_loss() const { return soft_loss_; }
89 [[nodiscard]] float hard_loss() const { return hard_loss_; }
91 [[nodiscard]] int64_t num_rows() const { return num_rows_; }
92
93 [[nodiscard]] const DistillationOptions& options() const { return options_; }
94
95private:
96 float Run(const Tensor& student_logits, const Tensor& teacher_logits, const Tensor* targets, float normalizer);
97
98 DeviceBackend* backend_;
99 DistillationOptions options_;
101 Tensor soft_grad_; // (kept rows, classes), already scaled by the normalizer
102 Tensor kept_; // (kept rows, 1) row indices, when rows were dropped
103 Shape logits_shape_ = Shape({0});
104 float soft_loss_ = 0.0f;
105 float hard_loss_ = 0.0f;
106 int64_t num_rows_ = 0;
107 bool all_rows_ = true;
108 bool has_targets_ = false;
109 bool has_forwarded_ = false;
110};
111
112} // namespace pulsatrix
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
Distillation loss between a student's and a teacher's logits, (..., classes): every row along the las...
Definition distillation_loss.hpp:58
float forward(const Tensor &student_logits, const Tensor &teacher_logits, const Tensor &targets, float normalizer=0.0f)
alpha · soft + (1 − alpha) · hard over the rows whose target isn't the ignore index.
float forward(const Tensor &student_logits, const Tensor &teacher_logits, float normalizer=0.0f)
The soft term alone, over every row.
const DistillationOptions & options() const
Definition distillation_loss.hpp:93
float soft_loss() const
The last forward()'s parts, each divided by the normalizer and unweighted: the soft term (T² · KL) an...
Definition distillation_loss.hpp:88
float hard_loss() const
Definition distillation_loss.hpp:89
Tensor backward() const
Gradient w.r.t. the student's logits, same shape; zero on ignored rows.
DistillationLoss(DeviceBackend *backend, DistillationOptions options={})
int64_t num_rows() const
Rows the last forward() kept.
Definition distillation_loss.hpp:91
An N-dimensional shape. A plain aggregate of dimensions with no invariant beyond "non-negative dimens...
Definition shape.hpp:24
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
sum over tokens of -log softmax(logits)[target], divided by a normalizer; tokens whose target is the ...
Definition token_cross_entropy_loss.hpp:28
static constexpr int64_t kIgnoreIndex
PyTorch's default ignore_index.
Definition token_cross_entropy_loss.hpp:31
Definition acquisition_functions.hpp:16
DistillationDivergence
Which way round the soft term's KL divergence goes.
Definition distillation_loss.hpp:33
@ Forward
KL(teacher ‖ student): the student covers every mode the teacher puts mass on.
@ Reverse
KL(student ‖ teacher): the student concentrates on the teacher's main modes.
DistillationLoss's settings.
Definition distillation_loss.hpp:39
DistillationDivergence divergence
Definition distillation_loss.hpp:45
float alpha
The soft term's weight when there are targets; the hard term gets 1 − alpha. Without targets the loss...
Definition distillation_loss.hpp:44
int64_t ignore_index
Rows whose target is this are left out of both terms (padding).
Definition distillation_loss.hpp:47
float temperature
Softens both distributions; 1 compares them as they are. Must be positive.
Definition distillation_loss.hpp:41
N-dimensional tensor – owns a buffer via DeviceBackend*, RAII (Rule of Five).
Cross-entropy over a batch of tokens, with padding ignored and a caller-chosen normalizer for exact g...