pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::HintLoss Class Reference

FitNets' hint loss: mean((r(student) − teacher)²), with r a learned linear map from the student's feature width to the teacher's. More...

#include <feature_distillation.hpp>

Public Member Functions

 HintLoss (int64_t student_width, int64_t teacher_width, DeviceBackend *backend, uint64_t seed=0)
 With a regressor from student_width to teacher_width.
 
 HintLoss (DeviceBackend *backend)
 Without a regressor: the student's features are compared as they are, so the widths must match.
 
float forward (const Tensor &student_features, const Tensor &teacher_features)
 Computes the loss and caches what backward() needs.
 
Tensor backward ()
 Gradient w.r.t. the student's features, their shape; also accumulates the regressor's parameter gradients.
 
LinearModule * regressor ()
 The regressor, or nullptr without one.
 

Detailed Description

FitNets' hint loss: mean((r(student) − teacher)²), with r a learned linear map from the student's feature width to the teacher's.

Note
Features are (..., width): every row along the last dimension is one feature vector, and the leading shapes must match.
The regressor is a LinearModule with a bias, started the way PyTorch starts nn.Linear (weights and bias uniform in ±1/√student_width) from seed: LinearModule itself starts at zero, which would pass no gradient back to the student. Step it with the student's optimizer, or its own: backward() accumulates its gradients.

Constructor & Destructor Documentation

◆ HintLoss() [1/2]

pulsatrix::HintLoss::HintLoss ( int64_t  student_width,
int64_t  teacher_width,
DeviceBackend *  backend,
uint64_t  seed = 0 
)

With a regressor from student_width to teacher_width.

◆ HintLoss() [2/2]

pulsatrix::HintLoss::HintLoss ( DeviceBackend *  backend)
explicit

Without a regressor: the student's features are compared as they are, so the widths must match.

Member Function Documentation

◆ backward()

Tensor pulsatrix::HintLoss::backward ( )

Gradient w.r.t. the student's features, their shape; also accumulates the regressor's parameter gradients.

Exceptions
std::logic_errorbefore forward().

◆ forward()

float pulsatrix::HintLoss::forward ( const Tensor &  student_features,
const Tensor &  teacher_features 
)

Computes the loss and caches what backward() needs.

Exceptions
std::invalid_argumentif a tensor's rank is below 2, the leading shapes differ, a width doesn't match the regressor's (or, without one, the two widths differ), or the tensors are on different devices.

◆ regressor()

LinearModule * pulsatrix::HintLoss::regressor ( )
inline

The regressor, or nullptr without one.


The documentation for this class was generated from the following file: