|
pulsatrix 1.1.0
|
DistilBERT's recipe for a smaller language model (Sanh et al., 2019; roadmap KD-4): a student with some of the teacher's layers, started from those layers' weights, and a cosine loss that aligns the two models' hidden states. More...
#include <cstdint>#include <memory>#include <vector>#include "pulsatrix/causal_lm.hpp"#include "pulsatrix/tensor.hpp"
Go to the source code of this file.
Classes | |
| class | pulsatrix::HiddenStateTap |
| An identity intervention that keeps the hidden states it sees and adds a gradient to the one flowing back through it: the way to train a CausalLM on a loss of a layer's output. More... | |
| class | pulsatrix::HiddenStateCosineLoss |
| DistilBERT's cosine loss: the mean over kept rows of 1 − cos(s, t), with s and t the student's and teacher's vectors, (..., width); PyTorch's CosineEmbeddingLoss with target 1. More... | |
Namespaces | |
| namespace | pulsatrix |
Functions | |
| std::vector< int64_t > | pulsatrix::EveryOtherLayer (int64_t num_layers) |
Layers 0, 2, 4, ... of a num_layers model: one out of two, DistilBERT's choice. | |
| std::unique_ptr< CausalLM > | pulsatrix::MakeStudentFromLayers (CausalLM &teacher, const std::vector< int64_t > &layers, DeviceBackend *backend) |
A CausalLM with teacher's config and weights but only the blocks in layers, in that order: the student's block j is a copy of the teacher's block layers[j]. The embedding, final norm and head are copied too; layer_types keeps the chosen layers'. | |
DistilBERT's recipe for a smaller language model (Sanh et al., 2019; roadmap KD-4): a student with some of the teacher's layers, started from those layers' weights, and a cosine loss that aligns the two models' hidden states.
DistilBERT keeps BERT's width and halves its depth. The student starts as a copy of the teacher with one layer out of two, so it begins close to the teacher, and trains on three losses:
DistilBERT weights them 5 : 2 : 1. In its ablation, the teacher's initialization and the distillation loss mattered most, and the language-modeling loss least.
CausalLM's backward() starts at the logits. To add the cosine loss's gradient at a layer's output, put a HiddenStateTap there with CausalLM::set_intervention(): it passes the hidden states through unchanged, keeps a copy of them, and adds the gradient it is given to the one backward() brings down.
In a pre-norm model such as Llama, the last layer's output is the residual stream before the final RMSNorm, not the normalized states DistilBERT's BERT returns. The cosine ignores each vector's length, but the stream's few large dimensions weigh on its direction.
tests/layer_student_test.cpp checks the cosine loss against tools/generate_layer_student_reference_values.py (numpy, gradients by central differences), and that a student with every layer computes the teacher's logits.