Deep Learning Modules and Layers¶
This is the core of pulsatrix: tensors, layers, optimizers and losses for building and training neural networks in C++. Start here to build a model. Every other section, from data loading to interpretability, builds on these types.
Everything is a concrete C++ type you can read and step into in a debugger:
- a
Tensorowns a buffer on a device through aDeviceBackend*(CPU, CUDA or HIP/ROCm); Modulesubclasses implement layers as plain tensor-in, tensor-out operations;- an opt-in
ComputationGraphandAutograd(automatic differentiation) recordNodes when you run modules throughModule::forward_traced(). The explainers use this path.
Every Module must implement an LRP (Layer-wise Relevance Propagation) rule, because
propagate_relevance() is pure virtual. All modules support the epsilon rule. LinearModule
and Conv2DModule also support the gamma, alpha-beta (ZPlus) and z-box rules. To run LRP over a
whole network, see Layer-wise Relevance Propagation.
What's inside¶
- Core:
Tensor,Shape,Module,ComputationGraph/Autograd,Node,OpType,DeviceBackend(CPUBackend/CUDABackend/HIPBackend) - Layers:
LinearModule,Conv2DModule,ReluModule,ActivationModule(ReLU, tanh, sigmoid, SiLU, exact GELU or GELU-tanh, chosen at construction),FlattenModule,SequentialModule,DropoutModule,EmbeddingModule,ResidualModule; normalization (LayerNormModule/RMSNormModule/GroupNormModule/BatchNormModule, withBatchNormFoldto fold an eval-mode BatchNorm into the layer before it); pooling (MaxPool2DModule/AvgPool2DModule/AdaptiveAvgPool2DModule).Conv2DModuletakes an optionalstrideand zeropadding, as intorch.nn.Conv2d, and every LRP rule handles both.MaxPool2DModulealso takes a stride and padding, so its windows may overlap (ResNet'sMaxPool2d(3, 2, 1)).ResidualModuletakes an optional shortcut module, such as ResNet's downsampling convolution. - Performance, with the same results:
Conv2DModulekeeps its unfolded input patches (im2col) only up to a workspace budget,set_max_workspace_bytes(), 16 MiB by default. Larger batches run in chunks of examples. Lower the budget if a large batch runs out of memory; the results don't change (HIP-7). On a GPU, fused kernels cut the launches: one for the optimizer, one for Linear plus ReLU inside aSequentialModule, and one batched GEMM per chunk in Conv2D's forward, backward and LRP (HIP-6). - Vision models:
TorchvisionResNetandTorchvisionVGGload torchvision's published ImageNet weights; ResNet18 and VGG16 are checked against PyTorch. See Vision models. - Sequence & attention:
RNNModule/LSTMModule/GRUModule,SoftmaxModule,RoPEModule,MultiHeadAttentionModule,SwiGLUModule,FeedForwardModule,TransformerBlock,EncoderBlock,MambaModule,RWKVModule,RetNetModule - Attention for pretrained LLMs:
MultiHeadAttentionModuleandTransformerBlocktake anAttentionConfigwith the settings Hugging Face checkpoints use: grouped-query attention (num_kv_heads), ahead_dimindependent ofd_model, optional projection biases, a causal mask, the RoPE pair layout (RoPELayout::RotateHalffor Llama-family models) and base (rope_theta), and QK-Norm.set_key_padding_mask()masks padding tokens andset_position_offset()shifts RoPE positions.LinearModulecan be built without a bias. The settings Gemma 3 adds (LLM-9) are: AttentionConfig::sliding_window, so each query sees only its last positionsscore_scale(Gemma'squery_pre_attn_scalar^-0.5)qk_norm_weight_offsetTransformerBlockOptions, which gives a GeGLU MLP (GatedActivation::GeluTanh), sandwich norms around attention and the MLP, and(1 + w)RMSNorms (RMSNormModule::set_weight_offset)TiedLMHeadModuleis the output layer of models withtie_word_embeddings: it computes logits with anEmbeddingModule's table, which stays a single parameter whose gradient sums both uses.- Text generation (
generation.hpp):Generate(model, prompt, config)decodes greedily or samples, with temperature, top-k, top-p, min-p, a repetition penalty and EOS tokens (Hugging Face'sGenerationConfignames and order). Sampling is seeded (config.seed, or the global seed).MakeNextTokenLogitsturns a module stack such as embedding → transformer blocks →TiedLMHeadModuleinto the model function. Each result also gives every new token's log-probability under the model. - KV cache (
kv_cache.hpp):MultiHeadAttentionModule::forward_cachedandTransformerBlock::forward_cachedprocess only new positions against a preallocatedKVCache.MakeCachedNextTokenLogits(embedding, blocks, head, backend, max_length)uses them for generation, reusing the cache for any prefix it has already seen; on CPU, 128 tokens from a 4-block, 128-wide model take 47 ms instead of 3.3 s. - Optimizers & losses:
SGDOptimizer(momentum, Nesterov),AdamOptimizer,AdamWOptimizer, all with parameter groups;MSELoss,CrossEntropyLoss,TokenCrossEntropyLoss,DistillationLoss,HiddenStateCosineLoss,BCEWithLogitsLoss,KLDivergenceLoss,CalibrationLoss - Generative building blocks:
Reparameterize(VAE),NoiseSchedule(linear and cosine schedules, DDPM and DDIM steps),SinusoidalTimestepEmbedding()andSampleContinuousDiffusion()(diffusion;trajectory_lrp.hppexplains its samples) - Training utilities:
ClipGradNorm,LRSchedule/LRScheduler, parameter groups (param_groups.hpp), checkpoints (SaveCheckpoint/LoadCheckpoint,safetensors.hpp),set_seed/set_deterministic, andMetricsSink/NoOpMetricsSink(wheretrain_steplogs its loss) - Selection:
top_k(), the k largest or smallest entries of every row along the last dimension, with their indices, on any device. NaN ranks above every number and ties keep the lower index first, so every backend selects the same entries in the same order. - Matrix decompositions (CPU, for matrices up to a few hundred wide):
SymmetricEigen,PowerIteration,QRandSVD, the building blocks for PCA, stable rank and orthonormal projections. Vector signs are fixed (each vector's largest entry is positive) and values come largest first, so the same matrix always gives the same answer.SymmetricEigenvaluesandSymmetricEigenInDoubletake and return doubles, for when the smallest eigenvalues matter. - Ready-made examples:
XorNetwork(a tiny MLP that learns XOR),MnistConvNet(a Conv2D MNIST classifier) andMnistIdxLoader(reads the MNIST IDX files)
The CPU backend is always built, and its large matrix multiplies use every core, splitting the
work so results are identical to a single-threaded run. To build the GPU backends, configure
CMake with -DPULSATRIX_ENABLE_CUDA=ON or -DPULSATRIX_ENABLE_HIP=ON.
Every layer and loss checks that the tensors it's given live on its own device, and throws
std::invalid_argument if not; move a tensor first with Tensor::to(). EmbeddingModule is the
exception: it accepts its indices from any device.
Full API reference: Doxygen: Deep Learning Modules and Layers
How to implement¶
Building and training a small network¶
A network is a plain C++ object with Module fields. Training is explicit, with no hidden
tape: you call forward() on each module, compute the loss, call each module's backward() in
reverse order, then call optimizer.step(module) for each layer with parameters.
XorNetwork packages that loop as train_step():
#include "pulsatrix/adam_optimizer.hpp"
#include "pulsatrix/cpu_backend.hpp"
#include "pulsatrix/metrics_sink.hpp"
#include "pulsatrix/xor_training_example.hpp"
using namespace pulsatrix;
CPUBackend backend;
XorNetwork net(&backend); // Linear(2,4) -> ReLU -> Linear(4,1)
AdamOptimizer optimizer(0.01f, &backend);
NoOpMetricsSink sink; // discards the logged loss
Tensor input(Shape({1, 2}), &backend, {1.0f, 0.0f});
Tensor target(Shape({1, 1}), &backend, {1.0f});
float loss = net.train_step(input, target, optimizer, sink, /*step=*/0);
Tensor pred = net.forward(input); // forward pass only
What's happening: XorNetwork owns two LinearModules and a ReluModule as plain
members (see include/pulsatrix/xor_training_example.hpp). forward() chains
linear1 → relu → linear2. train_step() does five things:
- zeroes the gradients;
- runs
forward(); - computes the
MSELoss; - calls
backward()on each layer in reverse order; - applies one Adam
step()perLinearModuleand logs the loss tosink.
Runnable version: examples/xor_demo.cpp
(CMake target xor_demo). See also the recipe.
Saving and loading weights¶
Checkpoints are safetensors files, the format most Hugging Face models ship in. It holds only numbers, so loading a file can't run code, unlike a PyTorch pickle.
#include "pulsatrix/checkpoint.hpp"
SaveCheckpoint("model.safetensors", model, optimizer); // also writes model.optim.safetensors
// ... later, or in another process:
LoadCheckpoint("model.safetensors", model, optimizer); // training resumes exactly
The optimizer state that can be saved is AdamOptimizer's or AdamWOptimizer's; SGD's momentum
buffers aren't saved yet.
- Every parameter and buffer (such as BatchNorm's running statistics) is stored under its
named_parameters()/named_buffers()name. Leave out the optimizer to save or load just the model. - A resumed run reproduces the original loss curve bit for bit.
- Loading is strict: a missing, unexpected or wrongly shaped entry throws, and nothing is changed.
Pass
CheckpointLoadOptions{false}to skip missing and unexpected names. Shapes are always checked. - Each file records a
format_version. Older versions load through a migration table, and newer ones are rejected. A plain safetensors file with matching names loads as version 0. - The optimizer file records which save of the model it belongs to, so stale optimizer state is refused.
The reader underneath (safetensors.hpp) treats every file as untrusted, and throws
std::invalid_argument for anything outside the format: offsets past the end of the file,
overlapping or missing byte ranges, sizes that overflow, and malformed headers. It's fuzzed under
AddressSanitizer (tools/fuzz/safetensors_fuzz.cpp). SafetensorsFile::Map memory-maps a file
instead of reading it, so only the parts you use are loaded: a 2.2 GB checkpoint opens in about a
millisecond.
Hugging Face checkpoints¶
hf_model.hpp reads a model directory as downloaded from the Hub.
#include "pulsatrix/hf_model.hpp"
HfModelConfig config = ReadHfConfig("SmolLM2-135M/config.json");
if (!config.unsupported.empty()) { /* features pulsatrix can't run yet, e.g. "sliding-window attention (LLM-9)" */ }
AttentionConfig attention = ToAttentionConfig(config); // GQA, rotate-half RoPE, biases, QK-Norm
HfCheckpoint weights = HfCheckpoint::Open("SmolLM2-135M"); // model.safetensors or its sharded index
for (const std::string& name : weights.names()) { /* weights.info(name), weights.tensor(name, &backend) */ }
HfModelConfigfills in what the file leaves to the model class:head_dimfrom the sizes, Qwen2's Q/K/V biases, Qwen3's and Gemma 3's QK-Norm, andeos_token_idas a list. It reads older (rope_scaling,torch_dtype) and newer (rope_parameters,dtype) field names, and a multimodal config'stext_config.unsupportedlists everything the config asks for that pulsatrix can't run yet, each with the roadmap item that adds it, including architectures that aren't causal language models. A loader should refuse a model whose list isn't empty rather than run it subtly wrong.HfCheckpointmemory-maps every shard. The index must match its shards exactly (every listed tensor present, nothing unlisted), and a shard must be a plain file name in the checkpoint directory, so an index can't point outside it.- Hub weights are usually bf16.
tensor()widens bf16, fp16 and both fp8 formats to fp32 exactly, and rounds fp64; integer tensors stay raw bytes (bytes()).
Loading a model. LoadCausalLM builds a CausalLM (embedding, transformer blocks, final
RMSNorm, tied or untied head: Llama, SmolLM2, Qwen2/2.5, Qwen3, Gemma 3) from a downloaded model directory
and loads its weights by name:
#include "pulsatrix/causal_lm.hpp"
std::unique_ptr<CausalLM> model = LoadCausalLM("SmolLM2-135M", &backend);
Tensor logits = model->forward(ids); // (1, L) token ids -> (1, L, vocab)
GenerationResult out = Generate(model->next_token_logits(512), prompt_ids, GenerationConfig{});
The real SmolLM2-135M loads in under a second, and its greedy continuations are identical to
transformers'. SmolLM2-135M, Qwen2.5-0.5B, Qwen3-0.6B and Llama-3.2-1B (with Llama 3's RoPE
scaling) all give logits within 1e-3 of transformers'. For tokenizing, generating and explaining
with these models end to end, see the Language Models guide. CausalLM is an ordinary module, so backward(),
propagate_relevance() and checkpoints work on it. Loading is strict: a config with features
pulsatrix can't run yet is refused, and every parameter must be loaded and every checkpoint
tensor used. LoadWeights with a WeightMapping manifest (source name, target name, a transpose,
an optional row slice for fused tensors) loads other layouts the same way.
PyTorch checkpoints (.pt, .pth)¶
pulsatrix never reads a PyTorch pickle file in C++. A .pt, .pth or .pkl file is a small
program, and loading one can run arbitrary code; malware has been found in published models, and
scanners that look for it get bypassed. Instead, convert the file to safetensors once, in Python,
with PyTorch's restricted loader:
pip install "torch>=2.6" safetensors
python3 tools/convert/pickle_to_safetensors.py resnet18-f37072fd.pth resnet18.safetensors
# wrote resnet18.safetensors: 102 tensors, 11,699,112 values, dtypes float32, verified
To load the output:
- torchvision's ResNet and VGG:
LoadTorchvisionWeights(see Vision models). - Any other model: read the tensors with
SafetensorsFile::Mapand copy them into your layers by name, or put the file in a directory asmodel.safetensorsand useHfCheckpoint::OpenwithLoadWeightsand aWeightMapping(as for Hugging Face checkpoints). PyTorch stores a Linear weight as (out, in) and pulsatrix as (in, out), so map each one withWeightTransform::Transpose. LoadCheckpointreads only pulsatrix's own checkpoints, and refuses a converted file.
What the converter does:
- Loads safely. It uses torch.load(weights_only=True), which only rebuilds tensors and
plain containers. It refuses torch older than 2.6, where weights_only could still run code
(CVE-2025-32434).
- Refuses what it can't load safely, with the reason:
- TorchScript archives;
- files that need any other Python object;
- sparse, quantized and complex tensors.
There is deliberately no unsafe mode. If a file you trust needs more, load it in a separate
environment and re-save model.state_dict() with torch.save.
- Finds the weights. A training checkpoint such as {"epoch": ..., "state_dict": {...},
"optimizer": ...} is unwrapped automatically, if exactly one of state_dict, model,
model_state_dict, module, net or ema holds tensors. Otherwise pass --key with a
dotted path. Nested dicts and lists become dotted names; numbers and strings are skipped and
listed.
- Options:
- --strip-prefix module. removes DataParallel's prefix.
- --float32 casts floating-point tensors to float32.
- --force allows overwriting the output; without it, the converter stops with an error if
the output exists.
- --no-verify skips reading the output back (see below).
- Writes clean tensors. Each tensor is made contiguous and gets its own bytes: tied weights
are written twice. Its dtype is kept: pulsatrix reads bf16 and fp16 by upcasting, and integer
buffers such as num_batches_tracked as raw bytes.
- Verifies. The source file's SHA-256 goes into the metadata, and the output is read back and
compared bit for bit.
The converted torchvision ResNet18 reads back in pulsatrix with all 11,699,112 values (its parameters and BatchNorm statistics) matching PyTorch's.
Vision models: ResNet and VGG¶
TorchvisionResNet and TorchvisionVGG (vision_models.hpp) rebuild torchvision's ResNet
with basic blocks and its VGG without BatchNorm from pulsatrix layers. They use torchvision's
parameter names, so the published ImageNet weights load once converted.
TorchvisionResNet::ResNet18() and TorchvisionVGG::VGG16() give those two models, and they are
the ones checked against PyTorch:
curl -LO https://download.pytorch.org/models/resnet18-f37072fd.pth
python3 tools/convert/pickle_to_safetensors.py resnet18-f37072fd.pth resnet18.safetensors
#include "pulsatrix/cpu_backend.hpp"
#include "pulsatrix/vision_models.hpp"
using namespace pulsatrix;
CPUBackend backend;
TorchvisionResNet model(TorchvisionResNet::ResNet18(), &backend); // or TorchvisionVGG::VGG16()
LoadTorchvisionWeights(model, "resnet18.safetensors"); // Linear weights are transposed for you
model.set_training(false); // eval mode: BatchNorm uses its running statistics, Dropout is off
Tensor logits = model.forward(x); // x: (N, 3, 224, 224), normalized with ImageNet's mean and std
- The input is torchvision's: resize the short side to 256, crop the center 224x224, scale
to [0, 1], subtract the mean (0.485, 0.456, 0.406) and divide by the standard deviation
(0.229, 0.224, 0.225), per channel.
ImageDecoder::DecodeFile(path, &backend, 3)reads a PNG or JPEG as (1, 3, H, W) in [0, 1]; pulsatrix has no resize or crop, so the ImageNet recipe does them by hand (ResizeAndCropin its source). - Other sizes come from the config,
ResNetConfig{blocks, width, num_classes}andVGGConfig{features, pool_size, hidden, num_classes}. ResNet34 isblocks = {3, 4, 6, 3}; VGG11, VGG13 and VGG19 are torchvision's configurations "A", "B" and "E" written asfeatures(each number a 3x3 convolution's width, 0 a 2x2 max pool). These load their published weights the same way, but only ResNet18 and VGG16 are tested on them. The tests also use a ResNet of width 4 and a VGG with 8 channels. - Matches PyTorch. On the published weights, the logits agree with torchvision's to 1e-4 (relative to the largest), on CPU and GPU.
To explain one, see LRP on ImageNet models: fold
the ResNet's BatchNorms, then explain through model.layers().
Reproducibility¶
Every random choice in pulsatrix comes from a seed, and nothing is seeded from the clock.
set_seed sets one global seed (default 0). Anything built without its own seed, such as a
DropoutModule, LinearProbe, SparseAutoencoder or a shuffling DataLoader, draws a distinct
seed from it in construction order:
#include "pulsatrix/determinism.hpp"
set_seed(1234); // same seed + same construction order = same weights, masks and shuffles
DropoutModule a(0.5f, &backend), b(0.5f, &backend); // different masks, both reproducible
DropoutModule c(0.5f, &backend, /*seed=*/7); // an explicit seed ignores the global one
Deterministic mode is on by default (set_deterministic(false) turns it off). pulsatrix's own
kernels use no atomics, so they always give the same result. In deterministic mode the GPU
backends also forbid atomics in hipBLAS and cuBLAS, so matrix products are reproducible too.
Random draws that use standard-library distributions (std::normal_distribution and friends)
are reproducible on one platform, but not between libstdc++ and MSVC.
Parameter groups and weight decay¶
SGDOptimizer and AdamOptimizer take a learning rate and weight decay per group of
parameters, chosen by name. Each parameter joins the first group that selects it; the rest use
the optimizer's own settings. The usual transformer recipe decays the weight matrices but not the
biases or normalization scales:
#include "pulsatrix/param_groups.hpp"
AdamOptimizer adam(3e-4f, &backend);
adam.set_weight_decay(0.01f); // the default group: every parameter no group selects
adam.set_param_groups({
{"no decay", param_select::one_dimensional(), 3e-4f, 0.0f}, // biases, norm scales
{"head", param_select::name_prefix("blocks.11"), 1e-3f, 0.01f}, // a faster last block
});
Weight decay works as in PyTorch's SGD and Adam: the step uses grad + weight_decay * value,
and the stored gradient is left unchanged. A group's learning_rate and weight_decay can be
changed between steps through param_groups().
For transformer fine-tuning, use AdamWOptimizer instead (decoupled weight decay, Loshchilov
and Hutter). It shrinks the weights directly, w <- (1 - lr * wd) * w, rather than adding the
decay to the gradient, where Adam's normalization would rescale it. Its default weight decay is
0.01, as in PyTorch. SGDOptimizer(lr, momentum, nesterov) adds momentum and Nesterov momentum,
with the same buffer rule as torch.optim.SGD.
Learning-rate schedules¶
LRSchedule gives a multiplier per step, with the formulas of Hugging Face's
get_*_schedule_with_warmup: Constant(warmup), Linear(warmup, total) and
Cosine(warmup, total, min_ratio). Warmup ramps from 0 to 1, then linear decays to 0 at total
and cosine to min_ratio. LRScheduler applies one to an optimizer: every rate, including each
parameter group's, becomes its starting rate times the multiplier.
#include "pulsatrix/lr_scheduler.hpp"
AdamWOptimizer opt(3e-4f, &backend);
LRScheduler scheduler(opt, LRSchedule::Cosine(/*warmup=*/100, /*total=*/10000));
for (int64_t step = 0; step < 10000; ++step) {
// ... forward, backward, ClipGradNorm ...
opt.step(model);
scheduler.step();
}
Set up parameter groups before building the scheduler. To resume, call
scheduler.set_last_step(n).
Token losses and gradient accumulation¶
TokenCrossEntropyLoss takes logits (..., classes) and integer targets with the same leading
shape, and skips targets equal to -100, PyTorch's padding convention. By default it averages
over the batch's real tokens.
To accumulate gradients over several micro-batches, don't average each micro-batch's own mean. When micro-batches hold different numbers of real tokens, that weights tokens unequally, which is the bug Hugging Face Trainer fixed in v4.46. Instead, divide every micro-batch by the token count of the whole window:
#include "pulsatrix/token_cross_entropy_loss.hpp"
int64_t total = 0;
for (const Tensor& t : window_targets) total += CountTargetTokens(t);
for (size_t k = 0; k < window_inputs.size(); ++k) {
Tensor logits = model.forward(window_inputs[k]);
(void)loss.forward(logits, window_targets[k], static_cast<float>(total));
(void)model.backward(loss.backward()); // gradients accumulate across micro-batches
}
(void)ClipGradNorm(model, 1.0f); // returns the norm before clipping
optimizer.step(model);
The accumulated gradients then equal those of one batch holding every micro-batch.
Knowledge distillation¶
DistillationLoss trains a student on a teacher's softened outputs (Hinton, Vinyals and Dean,
2015). Both sets of logits are divided by a temperature T before the softmax, which spreads the
teacher's probability over the classes it finds similar, and the soft term is T² times the KL
divergence between them. The T² keeps the soft term's gradient the same size as the hard term's
whatever T is. With targets, the loss is alpha times the soft term plus 1 - alpha times the
cross-entropy with the labels:
#include "pulsatrix/distillation_loss.hpp"
DistillationOptions o;
o.temperature = 2.0f; // the default; DistilBERT used 2
o.alpha = 0.5f; // soft term's weight, used only with targets
DistillationLoss loss(&backend, o);
Tensor teacher_logits = teacher.forward(inputs); // no backward through the teacher
float value = loss.forward(student.forward(inputs), teacher_logits, targets);
(void)student.backward(loss.backward());
optimizer.step(student);
- Shapes: logits are
(..., classes), likeTokenCrossEntropyLoss's, and the mean is over rows. For(N, classes)that is PyTorch'sF.kl_div(F.log_softmax(s / T, -1), F.softmax(t / T, -1), reduction="batchmean") * T * T. For a language model's(batch, length, vocab)it is the mean over tokens, which is what distilling a language model wants (batchmeanwould divide by the batch size alone). - Padding: rows whose target is
-100are left out of both terms. Without targets every row counts and the loss is the soft term alone. - Direction:
DistillationDivergence::Forward, KL(teacher ‖ student), is Hinton's: the student spreads its mass over everything the teacher considers.Reverse, KL(student ‖ teacher), is MiniLLM's choice for language models (Gu et al., ICLR 2024): the student settles on the teacher's main modes instead of covering the long tail it can't fit. - Accumulation: pass the window's row count as the normalizer, as for
TokenCrossEntropyLoss. soft_loss()andhard_loss()return the two terms of the lastforward(), unweighted.
The values and gradients are checked against a numpy implementation, with gradients by finite
differences (tools/generate_distillation_reference_values.py), and the HIP backend gives the
CPU's.
Distilling features¶
A student can also be taught what the teacher computes on the way to its output:
HintLoss(FitNets, Romero et al., ICLR 2015): the mean squared error between a student layer's features, mapped to the teacher's width by a learned linear regressor, and a teacher layer's. The regressor starts the way PyTorch'snn.Lineardoes (uniform in ±1/√width, from a seed).LinearModuleitself starts at zero, which would pass no gradient back to the student. Steploss.regressor()with an optimizer too.HintLoss(backend)compares features of the same width as they are.AttentionTransferLoss(Zagoruyko and Komodakis, ICLR 2017): for convolutional features(N, C, H, W), each example's spatial map, the mean over channels of the squared activation, normalized to unit length. The loss is the mean squared difference of the student's and teacher's maps, so the channel counts may differ, but the spatial sizes must match.AttentionMapreturns the map itself, for a figure.
Both return the gradient at the student's layer. To combine it with an output loss, run the student in two parts and add it where the parts meet:
#include "pulsatrix/feature_distillation.hpp"
SequentialModule front({&s1, &relu}); // the student up to the hinted layer
HintLoss hint(8, 16, &backend, /*seed=*/0);
Tensor hidden = front.forward(x);
float kd = distill.forward(head.forward(hidden), teacher_logits);
float h = hint.forward(hidden, teacher_hidden);
Tensor g = head.backward(distill.backward()); // the output loss's gradient at `hidden`
backend.axpby(1.0f, hint.backward().data(), 1.0f, g.data(), g.data(), static_cast<size_t>(g.numel()));
(void)front.backward(g);
Attention transfer is checked against the authors' code, and the hint loss against numpy, with
gradients by finite differences (tools/generate_feature_distillation_reference_values.py).
MiniLM's distillation of self-attention distributions isn't available: it needs a gradient
through MultiHeadAttentionModule's attention probabilities, which the module doesn't take.
Distilling into a soft decision tree¶
SoftDecisionTree (Frosst and Hinton, "Distilling a Neural Network Into a Soft Decision Tree",
2017) is a student you can read. It is a complete binary tree:
- Inner nodes: each one sends an input right with probability σ(β (x · wᵢ + bᵢ)), one learned linear filter per decision.
- Leaves: each holds a distribution over the classes.
- Output: the log of the mixture of the leaf distributions, weighted by each leaf's path
probability. So it trains with
DistillationLosson a teacher's logits, or withCrossEntropyLosson labels.
#include "pulsatrix/soft_decision_tree.hpp"
SoftDecisionTreeOptions o;
o.depth = 4; // 16 leaves
o.penalty = 0.01f; // Frosst and Hinton's balance penalty, λ
SoftDecisionTree tree(inputs, classes, &backend, o);
(void)loss.forward(tree.forward(x), teacher.forward(x));
(void)tree.backward(loss.backward()); // adds the penalty's gradient too
optimizer.step(tree);
// Explaining a prediction: the greedy path and what it tests.
(void)tree.forward(x);
int64_t leaf = tree.MostProbableLeaf()[0];
for (auto [node, right] : tree.PathTo(leaf)) { /* column `node` of tree.gates()'s weight */ }
- Penalty: it stops a node from sending almost everything one way. α is the share of the probability reaching the node that goes right, and the penalty adds −λ 2^(−depth) · ½ (log α + log(1 − α)) per node. Here α is computed over the current batch, not as the paper's moving average.
- Where it runs: the gates' linear layer runs on the backend's device, and the routing on the host. A 2-input, depth-3 tree distilled from a 2→16→3 network agrees with it on 125 of 128 held-out points in the tests.
- Checked against: a numpy implementation, with gradients by finite differences
(
tools/generate_soft_decision_tree_reference_values.py).
A smaller language model, DistilBERT's way¶
DistilBERT (Sanh et al., 2019) keeps its teacher's width, halves its depth, and starts the student
from the teacher's own weights, one layer out of two. MakeStudentFromLayers builds that student
from any CausalLM: block j of the student is a copy of the teacher's block layers[j], and the
embedding, final norm and head are copied too. A student given every layer computes exactly the
teacher's logits.
The student trains on three losses, which DistilBERT weights 5 : 2 : 1:
DistillationLosson the teacher's logits;- the language-modeling loss on the data;
HiddenStateCosineLoss: 1 − cos(h_s, h_t) between the two models' last hidden states, averaged over the tokens the mask keeps (PyTorch'sCosineEmbeddingLosswith target 1).
CausalLM::backward() starts at the logits, so the cosine loss's gradient goes in through a
HiddenStateTap, an identity intervention that keeps the states it sees and adds a gradient to
the one flowing back:
#include "pulsatrix/layer_student.hpp"
std::unique_ptr<CausalLM> student = MakeStudentFromLayers(*teacher, EveryOtherLayer(teacher->num_layers()), &backend);
const int64_t last = student->num_layers() - 1;
student->set_intervention(last, std::make_unique<HiddenStateTap>(&backend));
teacher->set_intervention(teacher->num_layers() - 1, std::make_unique<HiddenStateTap>(&backend));
auto* tap = static_cast<HiddenStateTap*>(student->intervention(last));
auto* teacher_tap = static_cast<HiddenStateTap*>(teacher->intervention(teacher->num_layers() - 1));
const Tensor teacher_logits = teacher->forward(ids);
const Tensor logits = student->forward(ids);
float kd = distill.forward(logits, teacher_logits); // DistillationLoss, alpha = 1
float lm = token_loss.forward(logits, targets); // TokenCrossEntropyLoss
float cos = cosine.forward(tap->hidden(), teacher_tap->hidden(), mask);
tap->add_gradient(cosine.backward()); // weight 1
Tensor g = distill.backward(); // then 5 · g + 2 · the token loss's gradient
backend.axpby(2.0f, token_loss.backward().data(), 5.0f, g.data(), g.data(), static_cast<size_t>(g.numel()));
(void)student->backward(g);
- Pre-norm models: in Llama and its relatives the last layer's output is the residual stream before the final RMSNorm, not the normalized states BERT returns. The cosine ignores the vectors' lengths, but the stream's few large dimensions weigh on their direction.
- Checked against: numpy for the cosine loss, with gradients by finite differences
(
tools/generate_layer_student_reference_values.py), and finite differences through a whole model for the tap. The HIP backend gives the CPU's values, and a student built on the GPU from a CPU teacher computes its logits.
The recipe Half the layers distills SmolLM2-135M-Instruct into 15 of its 30 blocks this way, against students trained without the teacher, without the cosine loss and from random blocks.
Born-again networks¶
A born-again network (Furlanello et al., ICML 2018) is a student the same size as its teacher,
trained on the teacher's softened outputs, which then teaches the next generation. Furlanello et
al. found students that beat their teachers and an ensemble of the generations that did better
still; later work found the gains small and not always there, so compare over several seeds. The
loop is ordinary training with DistillationLoss. born_again.hpp adds two pieces:
EnsembleLogProbabilities(logits, backend): log of the generations' mean distribution. It is the ensemble's prediction, and, given toDistillationLossas the teacher's logits, a teacher made of several generations.PermuteDarkKnowledge(teacher_logits, seed, backend): Furlanello et al.'s control. Each row keeps its largest logit in place and shuffles the rest among the other classes, so the teacher's top class and confidence stay and what it says about the other classes is scrambled. If a student taught this way does as well, the gain didn't come from that dark knowledge.
Their other control, which weights each example's label loss by the teacher's confidence, isn't available: it needs per-example weights in the cross-entropy loss.
The recipe Is the student born better? runs both on MNIST over five seeds, with the explanation test from KD-2.
Gradient clipping¶
ClipGradNorm(model, max_norm) scales all trainable gradients by one factor so their global L2
norm is at most max_norm, as torch.nn.utils.clip_grad_norm_ does. It returns the norm before
clipping, which is worth logging. Call it after accumulating gradients and before the optimizer
step:
#include "pulsatrix/grad_clipping.hpp"
const float norm = ClipGradNorm(model, 1.0f);
if (std::isfinite(norm)) {
optimizer.step(model); // a NaN or infinite norm leaves the gradients alone: skip this step
}
Freezing parameters¶
Every parameter has a name (named_parameters()), and set_requires_grad freezes or unfreezes
parameters by name. A name selects that parameter and everything under it:
TransformerBlock block(64, 4, 256, &backend);
block.set_requires_grad(false); // freeze everything
block.set_requires_grad(true, "mha.q_proj"); // then train only the query projection
A frozen parameter is never changed by any optimizer (SGDOptimizer, AdamOptimizer,
AdamWOptimizer), ClipGradNorm leaves it out, and backward() doesn't add to its gradient. The gradient passed back to the previous layer is exactly the same
as without freezing. LinearModule, Conv2DModule and EmbeddingModule skip computing a
frozen weight's gradient altogether, which is where fine-tuning saves time. A name that matches
nothing throws std::invalid_argument, so a typo can't leave the model silently trainable.
To train a small low-rank update instead of the weights themselves, see
Parameter-Efficient Fine-Tuning: ApplyLora() adds LoRA adapters and freezes
the rest.
Encoder layers¶
EncoderBlock is one transformer encoder layer whose layout is chosen with
EncoderBlockOptions:
norm:NormType::LayerNormorRMSNorm.norm_position:NormPosition::Pregivesx + f(norm(x))(ESM, GPT-2, Llama).Postgivesnorm(x + f(x))(BERT, the original Transformer).mlp:MlpType::PlainisFeedForwardModule: Linear → activation → Linear, as in BERT and ESM. Its default activation is exact GELU.GatedisSwiGLUModule: SwiGLU, or GeGLU withgate_activation.
Attention comes from the same AttentionConfig as TransformerBlock; an encoder leaves
causal off. The defaults are ESM-2's layer:
AttentionConfig attention;
attention.d_model = 320;
attention.num_heads = 20;
attention.rope_layout = RoPELayout::RotateHalf;
EncoderBlock esm_layer(attention, 1280, &backend); // pre-LayerNorm (eps 1e-5), GELU MLP
EncoderBlockOptions bert;
bert.norm_position = NormPosition::Post;
bert.norm_eps = 1e-12f;
attention.use_rope = false; // BERT adds learned positions before the first layer
EncoderBlock bert_layer(attention, 1280, &backend, bert);
- Checked against
transformers. The output and the input gradient match the library's ownEsmLayerandBertLayerto 2e-5 (tools/generate_encoder_reference.py). - The decoder layout reproduces
TransformerBlock. Pre-RMSNorm with a gated SiLU MLP givesTransformerBlock's output, gradients and relevance exactly. - Parameter names. Parameters are named
norm1.*,mha.*,norm2.*andmlp.*, soset_requires_gradand weight loading work the same way. - LRP.
- Each residual add uses the two-term epsilon rule.
- Norms and activations pass relevance through unchanged, as AttnLRP prescribes.
- Attention uses its AttnLRP rule.
- The result is linear in the incoming relevance in both layouts.
Choosing a normalization layer¶
LayerNormModule, RMSNormModule, GroupNormModule and BatchNormModule all implement the
same Module contract (forward(), backward(), propagate_relevance(),
named_parameters()). That makes them interchangeable in a SequentialModule. Pick one by
what it normalizes over:
LayerNormModuleandRMSNormModule: each row's features (the last dimension). RMSNorm skips the mean-centering.GroupNormModule: groups of channels, per example.BatchNormModule: each channel across the whole batch and spatial dimensions. In training mode it uses the current batch and updates running statistics (PyTorch's rule, momentum 0.1). Afterset_training(false)it uses the running statistics, so each sample's output no longer depends on the rest of its batch. Switch to eval mode before explaining a model.