90 bool has_hidden_ =
false;
91 bool has_extra_ =
false;
109 [[nodiscard]]
float forward(
const Tensor& student,
const Tensor& teacher,
const std::vector<float>& mask = {});
118 std::vector<double> grad_;
119 bool has_forwarded_ =
false;
A Llama-family decoder-only language model built from a Hugging Face config, and loading its Hugging ...
Token embedding, num_hidden_layers pre-norm transformer blocks, a final RMSNorm and a language-model ...
Definition causal_lm.hpp:82
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
virtual DeviceType device() const noexcept=0
Which device this backend's buffers reside on.
DistilBERT's cosine loss: the mean over kept rows of 1 โ cos(s, t), with s and t the student's and te...
Definition layer_student.hpp:100
HiddenStateCosineLoss(DeviceBackend *backend)
Tensor backward() const
Gradient with respect to the student's vectors, their shape.
float forward(const Tensor &student, const Tensor &teacher, const std::vector< float > &mask={})
An identity intervention that keeps the hidden states it sees and adds a gradient to the one flowing ...
Definition layer_student.hpp:65
const Tensor & hidden() const
The last forward()'s hidden states, (N, L, hidden_size).
OpType op_type() const override
This module's operation-category tag, for ComputationGraph node tagging.
Definition layer_student.hpp:75
Tensor backward(const Tensor &grad_output) override
grad_output plus the gradient given to add_gradient(), if any, which is then dropped.
HiddenStateTap(DeviceBackend *backend)
Tensor forward_at(const Tensor &hidden, int64_t start, int64_t prompt_length) override
The edit for hidden states covering positions [start, start + L) of a sequence whose prompt is prompt...
std::optional< DeviceType > compute_device() const override
The device this module computes on, so forward() can reject an input on another device before any ker...
Definition layer_student.hpp:76
void add_gradient(const Tensor &gradient)
A gradient for the next backward() to add, the hidden states' shape.
void release_activations() override
Frees the tensors forward() kept for backward() and the relevance passes; containers release their ch...
Tensor propagate_relevance(const Tensor &relevance_out, const LRPRuleConfig &config) override
Computes this module's contribution to LRP relevance propagation.
bool supports_lrp_rule(LRPRule) const override
Whether propagate_relevance() implements rule (no silent fallback: callers such as ExplainerContext::...
Definition layer_student.hpp:74
A trainable edit of the residual stream after a transformer block (TRN-9: LoReFT),...
Definition causal_lm.hpp:53
An N-dimensional shape. A plain aggregate of dimensions with no invariant beyond "non-negative dimens...
Definition shape.hpp:24
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) โ th...
Definition tensor.hpp:29
Definition acquisition_functions.hpp:16
std::vector< int64_t > EveryOtherLayer(int64_t num_layers)
Layers 0, 2, 4, ... of a num_layers model: one out of two, DistilBERT's choice.
DeviceType
Which physical device a Tensor's buffer resides on.
Definition device_backend.hpp:17
OpType
The op-type tag a Node carries. Charter Part 2 ยง3: nodes are tagged by a small closed set of op types...
Definition op_type.hpp:19
LRPRule
The LRP rule family a module applies. Semantics follow Zennit 1.0.0 exactly (Anders et al....
Definition lrp_rule_config.hpp:21
std::unique_ptr< CausalLM > MakeStudentFromLayers(CausalLM &teacher, const std::vector< int64_t > &layers, DeviceBackend *backend)
A CausalLM with teacher's config and weights but only the blocks in layers, in that order: the studen...
Configuration for LRP relevance propagation: which rule a module applies and its hyperparameters....
Definition lrp_rule_config.hpp:72
N-dimensional tensor โ owns a buffer via DeviceBackend*, RAII (Rule of Five).