pulsatrix 1.1.0
Loading...
Searching...
No Matches
feature_distillation.hpp
Go to the documentation of this file.
1
31#pragma once
32
33#include <cstdint>
34#include <memory>
35#include <vector>
36
39#include "pulsatrix/tensor.hpp"
40
41namespace pulsatrix {
42
53class HintLoss {
54public:
56 HintLoss(int64_t student_width, int64_t teacher_width, DeviceBackend* backend, uint64_t seed = 0);
59 explicit HintLoss(DeviceBackend* backend);
60
67 [[nodiscard]] float forward(const Tensor& student_features, const Tensor& teacher_features);
68
71 [[nodiscard]] Tensor backward();
72
74 [[nodiscard]] LinearModule* regressor() { return regressor_.get(); }
75
76private:
77 DeviceBackend* backend_;
78 std::unique_ptr<LinearModule> regressor_;
79 MSELoss mse_;
80 Shape student_shape_ = Shape({0});
81 bool has_forwarded_ = false;
82};
83
94public:
95 explicit AttentionTransferLoss(DeviceBackend* backend, double p = 2.0);
96
101 [[nodiscard]] float forward(const Tensor& student_features, const Tensor& teacher_features);
102
105 [[nodiscard]] Tensor backward() const;
106
107private:
108 DeviceBackend* backend_;
109 double p_;
110 Tensor student_;
111 std::vector<double> q_s_, q_t_, norms_;
112 int64_t batch_ = 0, channels_ = 0, positions_ = 0;
113 bool has_forwarded_ = false;
114};
115
121[[nodiscard]] Tensor AttentionMap(const Tensor& features, DeviceBackend* backend, double p = 2.0);
122
123} // namespace pulsatrix
Attention transfer: mean over examples and positions of (q_s − q_t)², where q is the L2-normalized ma...
Definition feature_distillation.hpp:93
Tensor backward() const
Gradient w.r.t. the student's features, their shape.
AttentionTransferLoss(DeviceBackend *backend, double p=2.0)
float forward(const Tensor &student_features, const Tensor &teacher_features)
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
FitNets' hint loss: mean((r(student) − teacher)²), with r a learned linear map from the student's fea...
Definition feature_distillation.hpp:53
float forward(const Tensor &student_features, const Tensor &teacher_features)
Computes the loss and caches what backward() needs.
HintLoss(DeviceBackend *backend)
Without a regressor: the student's features are compared as they are, so the widths must match.
LinearModule * regressor()
The regressor, or nullptr without one.
Definition feature_distillation.hpp:74
Tensor backward()
Gradient w.r.t. the student's features, their shape; also accumulates the regressor's parameter gradi...
HintLoss(int64_t student_width, int64_t teacher_width, DeviceBackend *backend, uint64_t seed=0)
With a regressor from student_width to teacher_width.
y = x @ W + b, batched (x is (N, in_features), y is (N, out_features)) – migrated from the original u...
Definition linear_module.hpp:43
MSE = mean((prediction - target)^2).
Definition mse_loss.hpp:19
An N-dimensional shape. A plain aggregate of dimensions with no invariant beyond "non-negative dimens...
Definition shape.hpp:24
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
Dense/fully-connected layer – the reference Module implementation.
Mean squared error loss.
Definition acquisition_functions.hpp:16
Tensor AttentionMap(const Tensor &features, DeviceBackend *backend, double p=2.0)
The attention map AttentionTransferLoss compares: (N, positions), each row L2-normalized,...
N-dimensional tensor – owns a buffer via DeviceBackend*, RAII (Rule of Five).