|
pulsatrix 1.1.0
|
Born-again networks (Furlanello et al., ICML 2018; roadmap KD-4): a student the size of its teacher, trained on the teacher's outputs, then a teacher for the next generation. More...

Go to the source code of this file.
Namespaces | |
| namespace | pulsatrix |
Functions | |
| Tensor | pulsatrix::EnsembleLogProbabilities (const std::vector< Tensor > &logits, DeviceBackend *backend) |
| log((1/G) Σ_g softmax(logits_g)) row by row, (..., classes), on the first tensor's device. | |
| Tensor | pulsatrix::PermuteDarkKnowledge (const Tensor &teacher_logits, uint64_t seed, DeviceBackend *backend) |
The teacher's logits with each row's non-maximal entries shuffled among themselves by a portable generator seeded with seed: the same result on every platform. The largest entry (the first, if tied) stays in place. | |
Born-again networks (Furlanello et al., ICML 2018; roadmap KD-4): a student the size of its teacher, trained on the teacher's outputs, then a teacher for the next generation.
Train a model on the labels; train a second with the same architecture on the first's softened outputs (DistillationLoss); repeat. Furlanello et al. found the students often beat their teachers, and an ensemble of the generations did better still. Later work found the gains small and not always there, so measure them over several seeds before believing them.
Two pieces here; the training loop is yours:
Not here: their other control, CWTM, which weights each example's label loss by the teacher's confidence. It needs per-example weights in the cross-entropy loss.