78 std::unique_ptr<LinearModule> regressor_;
81 bool has_forwarded_ =
false;
111 std::vector<double> q_s_, q_t_, norms_;
112 int64_t batch_ = 0, channels_ = 0, positions_ = 0;
113 bool has_forwarded_ =
false;
Attention transfer: mean over examples and positions of (q_s − q_t)², where q is the L2-normalized ma...
Definition feature_distillation.hpp:93
Tensor backward() const
Gradient w.r.t. the student's features, their shape.
AttentionTransferLoss(DeviceBackend *backend, double p=2.0)
float forward(const Tensor &student_features, const Tensor &teacher_features)
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
FitNets' hint loss: mean((r(student) − teacher)²), with r a learned linear map from the student's fea...
Definition feature_distillation.hpp:53
float forward(const Tensor &student_features, const Tensor &teacher_features)
Computes the loss and caches what backward() needs.
HintLoss(DeviceBackend *backend)
Without a regressor: the student's features are compared as they are, so the widths must match.
LinearModule * regressor()
The regressor, or nullptr without one.
Definition feature_distillation.hpp:74
Tensor backward()
Gradient w.r.t. the student's features, their shape; also accumulates the regressor's parameter gradi...
HintLoss(int64_t student_width, int64_t teacher_width, DeviceBackend *backend, uint64_t seed=0)
With a regressor from student_width to teacher_width.
y = x @ W + b, batched (x is (N, in_features), y is (N, out_features)) – migrated from the original u...
Definition linear_module.hpp:43
MSE = mean((prediction - target)^2).
Definition mse_loss.hpp:19
An N-dimensional shape. A plain aggregate of dimensions with no invariant beyond "non-negative dimens...
Definition shape.hpp:24
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
Dense/fully-connected layer – the reference Module implementation.
Definition acquisition_functions.hpp:16
Tensor AttentionMap(const Tensor &features, DeviceBackend *backend, double p=2.0)
The attention map AttentionTransferLoss compares: (N, positions), each row L2-normalized,...
N-dimensional tensor – owns a buffer via DeviceBackend*, RAII (Rule of Five).