FitNets' hint loss: mean((r(student) − teacher)²), with r a learned linear map from the student's feature width to the teacher's.
More...
#include <feature_distillation.hpp>
|
| | HintLoss (int64_t student_width, int64_t teacher_width, DeviceBackend *backend, uint64_t seed=0) |
| | With a regressor from student_width to teacher_width.
|
| |
| | HintLoss (DeviceBackend *backend) |
| | Without a regressor: the student's features are compared as they are, so the widths must match.
|
| |
| float | forward (const Tensor &student_features, const Tensor &teacher_features) |
| | Computes the loss and caches what backward() needs.
|
| |
| Tensor | backward () |
| | Gradient w.r.t. the student's features, their shape; also accumulates the regressor's parameter gradients.
|
| |
| LinearModule * | regressor () |
| | The regressor, or nullptr without one.
|
| |
FitNets' hint loss: mean((r(student) − teacher)²), with r a learned linear map from the student's feature width to the teacher's.
- Note
- Features are (..., width): every row along the last dimension is one feature vector, and the leading shapes must match.
-
The regressor is a LinearModule with a bias, started the way PyTorch starts nn.Linear (weights and bias uniform in ±1/√student_width) from
seed: LinearModule itself starts at zero, which would pass no gradient back to the student. Step it with the student's optimizer, or its own: backward() accumulates its gradients.
◆ HintLoss() [1/2]
| pulsatrix::HintLoss::HintLoss |
( |
int64_t |
student_width, |
|
|
int64_t |
teacher_width, |
|
|
DeviceBackend * |
backend, |
|
|
uint64_t |
seed = 0 |
|
) |
| |
With a regressor from student_width to teacher_width.
◆ HintLoss() [2/2]
Without a regressor: the student's features are compared as they are, so the widths must match.
◆ backward()
| Tensor pulsatrix::HintLoss::backward |
( |
| ) |
|
Gradient w.r.t. the student's features, their shape; also accumulates the regressor's parameter gradients.
- Exceptions
-
◆ forward()
| float pulsatrix::HintLoss::forward |
( |
const Tensor & |
student_features, |
|
|
const Tensor & |
teacher_features |
|
) |
| |
Computes the loss and caches what backward() needs.
- Exceptions
-
| std::invalid_argument | if a tensor's rank is below 2, the leading shapes differ, a width doesn't match the regressor's (or, without one, the two widths differ), or the tensors are on different devices. |
◆ regressor()
The regressor, or nullptr without one.
The documentation for this class was generated from the following file: