Attention transfer: mean over examples and positions of (q_s − q_t)², where q is the L2-normalized map mean_c |A_c|^p of each example's features.
More...
#include <feature_distillation.hpp>
Attention transfer: mean over examples and positions of (q_s − q_t)², where q is the L2-normalized map mean_c |A_c|^p of each example's features.
- Note
- Features are (N, C, ...): the dimensions after the channels are flattened into positions. Computed on the host in double precision, whatever the tensors' device; the gradient is returned on the student features' device.
-
p = 2, the default and the paper's choice, is the authors'
x.pow(2).mean(1); other p use |A|^p, the paper's definition.
◆ AttentionTransferLoss()
| pulsatrix::AttentionTransferLoss::AttentionTransferLoss |
( |
DeviceBackend * |
backend, |
|
|
double |
p = 2.0 |
|
) |
| |
|
explicit |
◆ backward()
| Tensor pulsatrix::AttentionTransferLoss::backward |
( |
| ) |
const |
Gradient w.r.t. the student's features, their shape.
- Exceptions
-
◆ forward()
| float pulsatrix::AttentionTransferLoss::forward |
( |
const Tensor & |
student_features, |
|
|
const Tensor & |
teacher_features |
|
) |
| |
- Exceptions
-
| std::invalid_argument | if a tensor's rank is below 3, the batch sizes or numbers of positions differ, or the tensors are on different devices. |
The documentation for this class was generated from the following file: