pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::AttentionTransferLoss Class Reference

Attention transfer: mean over examples and positions of (q_s − q_t)², where q is the L2-normalized map mean_c |A_c|^p of each example's features. More...

#include <feature_distillation.hpp>

Public Member Functions

 AttentionTransferLoss (DeviceBackend *backend, double p=2.0)
 
float forward (const Tensor &student_features, const Tensor &teacher_features)
 
Tensor backward () const
 Gradient w.r.t. the student's features, their shape.
 

Detailed Description

Attention transfer: mean over examples and positions of (q_s − q_t)², where q is the L2-normalized map mean_c |A_c|^p of each example's features.

Note
Features are (N, C, ...): the dimensions after the channels are flattened into positions. Computed on the host in double precision, whatever the tensors' device; the gradient is returned on the student features' device.
p = 2, the default and the paper's choice, is the authors' x.pow(2).mean(1); other p use |A|^p, the paper's definition.

Constructor & Destructor Documentation

◆ AttentionTransferLoss()

pulsatrix::AttentionTransferLoss::AttentionTransferLoss ( DeviceBackend *  backend,
double  p = 2.0 
)
explicit

Member Function Documentation

◆ backward()

Tensor pulsatrix::AttentionTransferLoss::backward ( ) const

Gradient w.r.t. the student's features, their shape.

Exceptions
std::logic_errorbefore forward().

◆ forward()

float pulsatrix::AttentionTransferLoss::forward ( const Tensor &  student_features,
const Tensor &  teacher_features 
)
Exceptions
std::invalid_argumentif a tensor's rank is below 3, the batch sizes or numbers of positions differ, or the tensors are on different devices.

The documentation for this class was generated from the following file: