pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::DistillationLoss Class Reference

Distillation loss between a student's and a teacher's logits, (..., classes): every row along the last dimension is one distribution. More...

#include <distillation_loss.hpp>

Public Member Functions

 DistillationLoss (DeviceBackend *backend, DistillationOptions options={})
 
float forward (const Tensor &student_logits, const Tensor &teacher_logits, float normalizer=0.0f)
 The soft term alone, over every row.
 
float forward (const Tensor &student_logits, const Tensor &teacher_logits, const Tensor &targets, float normalizer=0.0f)
 alpha · soft + (1 − alpha) · hard over the rows whose target isn't the ignore index.
 
Tensor backward () const
 Gradient w.r.t. the student's logits, same shape; zero on ignored rows.
 
float soft_loss () const
 The last forward()'s parts, each divided by the normalizer and unweighted: the soft term (T² · KL) and the hard term (0 without targets).
 
float hard_loss () const
 
int64_t num_rows () const
 Rows the last forward() kept.
 
const DistillationOptions & options () const
 

Detailed Description

Distillation loss between a student's and a teacher's logits, (..., classes): every row along the last dimension is one distribution.

Note
Device-generic: everything runs on the logits' device through softmax_rows, logsumexp_rows and axpby. forward() reads back the loss's parts; with targets it also reads the targets, one float per row, to find the rows to keep.
The teacher's logits get no gradient: the teacher is fixed.

Constructor & Destructor Documentation

◆ DistillationLoss()

pulsatrix::DistillationLoss::DistillationLoss ( DeviceBackend *  backend,
DistillationOptions  options = {} 
)
explicit

Member Function Documentation

◆ backward()

Tensor pulsatrix::DistillationLoss::backward ( ) const

Gradient w.r.t. the student's logits, same shape; zero on ignored rows.

Exceptions
std::logic_errorbefore forward().

◆ forward() [1/2]

float pulsatrix::DistillationLoss::forward ( const Tensor &  student_logits,
const Tensor &  teacher_logits,
const Tensor &  targets,
float  normalizer = 0.0f 
)

alpha · soft + (1 − alpha) · hard over the rows whose target isn't the ignore index.

Parameters
targetsThe logits' leading shape, whole-number floats, as for TokenCrossEntropyLoss.
normalizer0 (the default) means the number of kept rows.
Exceptions
std::invalid_argumentas the other overload, or for targets TokenCrossEntropyLoss rejects.

◆ forward() [2/2]

float pulsatrix::DistillationLoss::forward ( const Tensor &  student_logits,
const Tensor &  teacher_logits,
float  normalizer = 0.0f 
)

The soft term alone, over every row.

Parameters
normalizerWhat the summed loss is divided by; 0 (the default) means the number of rows. Pass the whole window's row count to accumulate gradients over micro-batches.
Exceptions
std::invalid_argumentif the logits aren't (..., classes) with rank >= 2, the teacher's shape isn't the student's, they're on different devices, or the normalizer is negative or not finite.

◆ hard_loss()

float pulsatrix::DistillationLoss::hard_loss ( ) const
inline

◆ num_rows()

int64_t pulsatrix::DistillationLoss::num_rows ( ) const
inline

Rows the last forward() kept.

◆ options()

const DistillationOptions & pulsatrix::DistillationLoss::options ( ) const
inline

◆ soft_loss()

float pulsatrix::DistillationLoss::soft_loss ( ) const
inline

The last forward()'s parts, each divided by the normalizer and unweighted: the soft term (T² · KL) and the hard term (0 without targets).


The documentation for this class was generated from the following file: