70 [[nodiscard]]
float forward(
const Tensor& student_logits,
const Tensor& teacher_logits,
float normalizer = 0.0f);
80 float normalizer = 0.0f);
88 [[nodiscard]]
float soft_loss()
const {
return soft_loss_; }
89 [[nodiscard]]
float hard_loss()
const {
return hard_loss_; }
91 [[nodiscard]] int64_t
num_rows()
const {
return num_rows_; }
96 float Run(
const Tensor& student_logits,
const Tensor& teacher_logits,
const Tensor* targets,
float normalizer);
104 float soft_loss_ = 0.0f;
105 float hard_loss_ = 0.0f;
106 int64_t num_rows_ = 0;
107 bool all_rows_ =
true;
108 bool has_targets_ =
false;
109 bool has_forwarded_ =
false;
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
Distillation loss between a student's and a teacher's logits, (..., classes): every row along the las...
Definition distillation_loss.hpp:58
float forward(const Tensor &student_logits, const Tensor &teacher_logits, const Tensor &targets, float normalizer=0.0f)
alpha · soft + (1 − alpha) · hard over the rows whose target isn't the ignore index.
float forward(const Tensor &student_logits, const Tensor &teacher_logits, float normalizer=0.0f)
The soft term alone, over every row.
const DistillationOptions & options() const
Definition distillation_loss.hpp:93
float soft_loss() const
The last forward()'s parts, each divided by the normalizer and unweighted: the soft term (T² · KL) an...
Definition distillation_loss.hpp:88
float hard_loss() const
Definition distillation_loss.hpp:89
Tensor backward() const
Gradient w.r.t. the student's logits, same shape; zero on ignored rows.
DistillationLoss(DeviceBackend *backend, DistillationOptions options={})
int64_t num_rows() const
Rows the last forward() kept.
Definition distillation_loss.hpp:91
An N-dimensional shape. A plain aggregate of dimensions with no invariant beyond "non-negative dimens...
Definition shape.hpp:24
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
sum over tokens of -log softmax(logits)[target], divided by a normalizer; tokens whose target is the ...
Definition token_cross_entropy_loss.hpp:28
static constexpr int64_t kIgnoreIndex
PyTorch's default ignore_index.
Definition token_cross_entropy_loss.hpp:31
Definition acquisition_functions.hpp:16
DistillationDivergence
Which way round the soft term's KL divergence goes.
Definition distillation_loss.hpp:33
@ Forward
KL(teacher ‖ student): the student covers every mode the teacher puts mass on.
@ Reverse
KL(student ‖ teacher): the student concentrates on the teacher's main modes.
DistillationLoss's settings.
Definition distillation_loss.hpp:39
DistillationDivergence divergence
Definition distillation_loss.hpp:45
float alpha
The soft term's weight when there are targets; the hard term gets 1 − alpha. Without targets the loss...
Definition distillation_loss.hpp:44
int64_t ignore_index
Rows whose target is this are left out of both terms (padding).
Definition distillation_loss.hpp:47
float temperature
Softens both distributions; 1 compares them as they are. Must be positive.
Definition distillation_loss.hpp:41
N-dimensional tensor – owns a buffer via DeviceBackend*, RAII (Rule of Five).
Cross-entropy over a batch of tokens, with padding ignored and a caller-chosen normalizer for exact g...