|
pulsatrix 1.1.0
|
Distilling a teacher's intermediate features, not only its outputs (roadmap KD-3): FitNets' hint loss (Romero et al., ICLR 2015) and attention transfer (Zagoruyko and Komodakis, ICLR 2017). More...
#include <cstdint>#include <memory>#include <vector>#include "pulsatrix/linear_module.hpp"#include "pulsatrix/mse_loss.hpp"#include "pulsatrix/tensor.hpp"
Go to the source code of this file.
Classes | |
| class | pulsatrix::HintLoss |
| FitNets' hint loss: mean((r(student) − teacher)²), with r a learned linear map from the student's feature width to the teacher's. More... | |
| class | pulsatrix::AttentionTransferLoss |
| Attention transfer: mean over examples and positions of (q_s − q_t)², where q is the L2-normalized map mean_c |A_c|^p of each example's features. More... | |
Namespaces | |
| namespace | pulsatrix |
Functions | |
| Tensor | pulsatrix::AttentionMap (const Tensor &features, DeviceBackend *backend, double p=2.0) |
The attention map AttentionTransferLoss compares: (N, positions), each row L2-normalized, on features' device. | |
Distilling a teacher's intermediate features, not only its outputs (roadmap KD-3): FitNets' hint loss (Romero et al., ICLR 2015) and attention transfer (Zagoruyko and Komodakis, ICLR 2017).
DistillationLoss matches a student's outputs to its teacher's. These losses match what happens on the way:
Both losses return the gradient with respect to the student's features, at the layer they come from. To train through it, run the student in two parts (two SequentialModules over its layers, say) and add this gradient to the one the second part's backward() returns before calling the first part's backward().
tests/feature_distillation_test.cpp checks both against tools/generate_feature_distillation_reference_values.py, which writes attention transfer the way the authors' code does and takes gradients by central differences.