DistilBERT's cosine loss: the mean over kept rows of 1 − cos(s, t), with s and t the student's and teacher's vectors, (..., width); PyTorch's CosineEmbeddingLoss with target 1.
More...
#include <layer_student.hpp>
DistilBERT's cosine loss: the mean over kept rows of 1 − cos(s, t), with s and t the student's and teacher's vectors, (..., width); PyTorch's CosineEmbeddingLoss with target 1.
- Note
- Computed on the host in double precision; the gradient comes back on the student's device. A row whose student or teacher vector is zero counts as cos = 0, with no gradient.
◆ HiddenStateCosineLoss()
| pulsatrix::HiddenStateCosineLoss::HiddenStateCosineLoss |
( |
DeviceBackend * |
backend | ) |
|
|
explicit |
◆ backward()
| Tensor pulsatrix::HiddenStateCosineLoss::backward |
( |
| ) |
const |
Gradient with respect to the student's vectors, their shape.
- Exceptions
-
◆ forward()
| float pulsatrix::HiddenStateCosineLoss::forward |
( |
const Tensor & |
student, |
|
|
const Tensor & |
teacher, |
|
|
const std::vector< float > & |
mask = {} |
|
) |
| |
- Parameters
-
| mask | One value per row, 0 to leave it out (padding) and anything else to keep it; empty keeps every row. |
- Exceptions
-
| std::invalid_argument | if the shapes differ, the rank is below 2, the tensors are on different devices, the mask has the wrong length, or it keeps no row. |
The documentation for this class was generated from the following file: