|
pulsatrix 1.1.0
|
Knowledge distillation (roadmap KD-1): a student learns from a teacher's softened outputs (Hinton, Vinyals and Dean, 2015), optionally mixed with the hard labels. More...
#include <cstdint>#include "pulsatrix/tensor.hpp"#include "pulsatrix/token_cross_entropy_loss.hpp"
Go to the source code of this file.
Classes | |
| struct | pulsatrix::DistillationOptions |
| DistillationLoss's settings. More... | |
| class | pulsatrix::DistillationLoss |
| Distillation loss between a student's and a teacher's logits, (..., classes): every row along the last dimension is one distribution. More... | |
Namespaces | |
| namespace | pulsatrix |
Enumerations | |
| enum class | pulsatrix::DistillationDivergence { pulsatrix::Forward , pulsatrix::Reverse } |
| Which way round the soft term's KL divergence goes. More... | |
Knowledge distillation (roadmap KD-1): a student learns from a teacher's softened outputs (Hinton, Vinyals and Dean, 2015), optionally mixed with the hard labels.
With temperature T, student logits s, teacher logits t and p = softmax(·/T), each row (one example, or one token) contributes
soft = T² · KL(p_t ‖ p_s) (forward KL, Hinton's), or soft = T² · KL(p_s ‖ p_t) (reverse KL, MiniLLM's mode-seeking choice for LLMs),
and with targets the loss is alpha · soft + (1 − alpha) · cross-entropy(s, target) at T = 1. The T² keeps the soft term's gradient the same size as the hard term's as T changes: the gradient of KL with respect to s is (p_s − p_t) / T, so T² · KL's is T · (p_s − p_t).
The forward soft term with the mean over rows is PyTorch's F.kl_div(F.log_softmax(s / T, -1), F.softmax(t / T, -1), reduction="batchmean") * T * T for (N, classes) logits. For (batch, length, classes) logits the mean is over tokens, not sequences, which is what distillation of language models does (batchmean would divide by the batch size alone). tests/distillation_loss_test.cpp checks values and gradients against tools/generate_distillation_reference_values.py.