Distillation loss between a student's and a teacher's logits, (..., classes): every row along the last dimension is one distribution.
More...
#include <distillation_loss.hpp>
Distillation loss between a student's and a teacher's logits, (..., classes): every row along the last dimension is one distribution.
- Note
- Device-generic: everything runs on the logits' device through softmax_rows, logsumexp_rows and axpby. forward() reads back the loss's parts; with targets it also reads the targets, one float per row, to find the rows to keep.
-
The teacher's logits get no gradient: the teacher is fixed.
◆ DistillationLoss()
◆ backward()
| Tensor pulsatrix::DistillationLoss::backward |
( |
| ) |
const |
Gradient w.r.t. the student's logits, same shape; zero on ignored rows.
- Exceptions
-
◆ forward() [1/2]
| float pulsatrix::DistillationLoss::forward |
( |
const Tensor & |
student_logits, |
|
|
const Tensor & |
teacher_logits, |
|
|
const Tensor & |
targets, |
|
|
float |
normalizer = 0.0f |
|
) |
| |
alpha · soft + (1 − alpha) · hard over the rows whose target isn't the ignore index.
- Parameters
-
| targets | The logits' leading shape, whole-number floats, as for TokenCrossEntropyLoss. |
| normalizer | 0 (the default) means the number of kept rows. |
- Exceptions
-
◆ forward() [2/2]
| float pulsatrix::DistillationLoss::forward |
( |
const Tensor & |
student_logits, |
|
|
const Tensor & |
teacher_logits, |
|
|
float |
normalizer = 0.0f |
|
) |
| |
The soft term alone, over every row.
- Parameters
-
| normalizer | What the summed loss is divided by; 0 (the default) means the number of rows. Pass the whole window's row count to accumulate gradients over micro-batches. |
- Exceptions
-
| std::invalid_argument | if the logits aren't (..., classes) with rank >= 2, the teacher's shape isn't the student's, they're on different devices, or the normalizer is negative or not finite. |
◆ hard_loss()
| float pulsatrix::DistillationLoss::hard_loss |
( |
| ) |
const |
|
inline |
◆ num_rows()
| int64_t pulsatrix::DistillationLoss::num_rows |
( |
| ) |
const |
|
inline |
◆ options()
◆ soft_loss()
| float pulsatrix::DistillationLoss::soft_loss |
( |
| ) |
const |
|
inline |
The last forward()'s parts, each divided by the normalizer and unweighted: the soft term (T² · KL) and the hard term (0 without targets).
The documentation for this class was generated from the following file: