pulsatrix 1.1.0
Loading...
Searching...
No Matches
distillation_loss.hpp File Reference

Knowledge distillation (roadmap KD-1): a student learns from a teacher's softened outputs (Hinton, Vinyals and Dean, 2015), optionally mixed with the hard labels. More...

#include <cstdint>
#include "pulsatrix/tensor.hpp"
#include "pulsatrix/token_cross_entropy_loss.hpp"
Include dependency graph for distillation_loss.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::DistillationOptions
 DistillationLoss's settings. More...
 
class  pulsatrix::DistillationLoss
 Distillation loss between a student's and a teacher's logits, (..., classes): every row along the last dimension is one distribution. More...
 

Namespaces

namespace  pulsatrix
 

Enumerations

enum class  pulsatrix::DistillationDivergence { pulsatrix::Forward , pulsatrix::Reverse }
 Which way round the soft term's KL divergence goes. More...
 

Detailed Description

Knowledge distillation (roadmap KD-1): a student learns from a teacher's softened outputs (Hinton, Vinyals and Dean, 2015), optionally mixed with the hard labels.

With temperature T, student logits s, teacher logits t and p = softmax(·/T), each row (one example, or one token) contributes

soft = T² · KL(p_t ‖ p_s) (forward KL, Hinton's), or soft = T² · KL(p_s ‖ p_t) (reverse KL, MiniLLM's mode-seeking choice for LLMs),

and with targets the loss is alpha · soft + (1 − alpha) · cross-entropy(s, target) at T = 1. The T² keeps the soft term's gradient the same size as the hard term's as T changes: the gradient of KL with respect to s is (p_s − p_t) / T, so T² · KL's is T · (p_s − p_t).

The forward soft term with the mean over rows is PyTorch's F.kl_div(F.log_softmax(s / T, -1), F.softmax(t / T, -1), reduction="batchmean") * T * T for (N, classes) logits. For (batch, length, classes) logits the mean is over tokens, not sequences, which is what distillation of language models does (batchmean would divide by the batch size alone). tests/distillation_loss_test.cpp checks values and gradients against tools/generate_distillation_reference_values.py.