pulsatrix 1.1.0
Loading...
Searching...
No Matches
feature_distillation.hpp File Reference

Distilling a teacher's intermediate features, not only its outputs (roadmap KD-3): FitNets' hint loss (Romero et al., ICLR 2015) and attention transfer (Zagoruyko and Komodakis, ICLR 2017). More...

#include <cstdint>
#include <memory>
#include <vector>
#include "pulsatrix/linear_module.hpp"
#include "pulsatrix/mse_loss.hpp"
#include "pulsatrix/tensor.hpp"
Include dependency graph for feature_distillation.hpp:

Go to the source code of this file.

Classes

class  pulsatrix::HintLoss
 FitNets' hint loss: mean((r(student) − teacher)²), with r a learned linear map from the student's feature width to the teacher's. More...
 
class  pulsatrix::AttentionTransferLoss
 Attention transfer: mean over examples and positions of (q_s − q_t)², where q is the L2-normalized map mean_c |A_c|^p of each example's features. More...
 

Namespaces

namespace  pulsatrix
 

Functions

Tensor pulsatrix::AttentionMap (const Tensor &features, DeviceBackend *backend, double p=2.0)
 The attention map AttentionTransferLoss compares: (N, positions), each row L2-normalized, on features' device.
 

Detailed Description

Distilling a teacher's intermediate features, not only its outputs (roadmap KD-3): FitNets' hint loss (Romero et al., ICLR 2015) and attention transfer (Zagoruyko and Komodakis, ICLR 2017).

DistillationLoss matches a student's outputs to its teacher's. These losses match what happens on the way:

  • HintLoss: the mean squared error between a student layer's features, mapped to the teacher's width by a learned linear regressor, and a teacher layer's features. FitNets trains the student up to that layer this way first, then the whole student with DistillationLoss.
  • AttentionTransferLoss: for convolutional features (N, C, H, W), each layer's spatial attention map Q = mean over channels of |A|^p, flattened and L2-normalized per example; the loss is the mean squared difference of the student's and teacher's maps. The channel counts may differ; the spatial sizes must match.

Both losses return the gradient with respect to the student's features, at the layer they come from. To train through it, run the student in two parts (two SequentialModules over its layers, say) and add this gradient to the one the second part's backward() returns before calling the first part's backward().

tests/feature_distillation_test.cpp checks both against tools/generate_feature_distillation_reference_values.py, which writes attention transfer the way the authors' code does and takes gradients by central differences.

Note
Not here: MiniLM's distillation of self-attention distributions. It needs the gradient of the loss to flow back through MultiHeadAttentionModule's attention probabilities, which the module doesn't take.