pulsatrix 1.1.0
Loading...
Searching...
No Matches
layer_student.hpp File Reference

DistilBERT's recipe for a smaller language model (Sanh et al., 2019; roadmap KD-4): a student with some of the teacher's layers, started from those layers' weights, and a cosine loss that aligns the two models' hidden states. More...

#include <cstdint>
#include <memory>
#include <vector>
#include "pulsatrix/causal_lm.hpp"
#include "pulsatrix/tensor.hpp"
Include dependency graph for layer_student.hpp:

Go to the source code of this file.

Classes

class  pulsatrix::HiddenStateTap
 An identity intervention that keeps the hidden states it sees and adds a gradient to the one flowing back through it: the way to train a CausalLM on a loss of a layer's output. More...
 
class  pulsatrix::HiddenStateCosineLoss
 DistilBERT's cosine loss: the mean over kept rows of 1 − cos(s, t), with s and t the student's and teacher's vectors, (..., width); PyTorch's CosineEmbeddingLoss with target 1. More...
 

Namespaces

namespace  pulsatrix
 

Functions

std::vector< int64_t > pulsatrix::EveryOtherLayer (int64_t num_layers)
 Layers 0, 2, 4, ... of a num_layers model: one out of two, DistilBERT's choice.
 
std::unique_ptr< CausalLM > pulsatrix::MakeStudentFromLayers (CausalLM &teacher, const std::vector< int64_t > &layers, DeviceBackend *backend)
 A CausalLM with teacher's config and weights but only the blocks in layers, in that order: the student's block j is a copy of the teacher's block layers[j]. The embedding, final norm and head are copied too; layer_types keeps the chosen layers'.
 

Detailed Description

DistilBERT's recipe for a smaller language model (Sanh et al., 2019; roadmap KD-4): a student with some of the teacher's layers, started from those layers' weights, and a cosine loss that aligns the two models' hidden states.

DistilBERT keeps BERT's width and halves its depth. The student starts as a copy of the teacher with one layer out of two, so it begins close to the teacher, and trains on three losses:

  • the distillation loss on the teacher's softened outputs (DistillationLoss);
  • the usual language-modeling loss on the data (TokenCrossEntropyLoss, or DistillationLoss's hard term);
  • a cosine loss, 1 − cos(h_s, h_t), between the two models' last hidden states, averaged over tokens (HiddenStateCosineLoss), which pulls the student's states toward the teacher's.

DistilBERT weights them 5 : 2 : 1. In its ablation, the teacher's initialization and the distillation loss mattered most, and the language-modeling loss least.

CausalLM's backward() starts at the logits. To add the cosine loss's gradient at a layer's output, put a HiddenStateTap there with CausalLM::set_intervention(): it passes the hidden states through unchanged, keeps a copy of them, and adds the gradient it is given to the one backward() brings down.

In a pre-norm model such as Llama, the last layer's output is the residual stream before the final RMSNorm, not the normalized states DistilBERT's BERT returns. The cosine ignores each vector's length, but the stream's few large dimensions weigh on its direction.

tests/layer_student_test.cpp checks the cosine loss against tools/generate_layer_student_reference_values.py (numpy, gradients by central differences), and that a student with every layer computes the teacher's logits.