pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::HiddenStateCosineLoss Class Reference

DistilBERT's cosine loss: the mean over kept rows of 1 − cos(s, t), with s and t the student's and teacher's vectors, (..., width); PyTorch's CosineEmbeddingLoss with target 1. More...

#include <layer_student.hpp>

Public Member Functions

 HiddenStateCosineLoss (DeviceBackend *backend)
 
float forward (const Tensor &student, const Tensor &teacher, const std::vector< float > &mask={})
 
Tensor backward () const
 Gradient with respect to the student's vectors, their shape.
 

Detailed Description

DistilBERT's cosine loss: the mean over kept rows of 1 − cos(s, t), with s and t the student's and teacher's vectors, (..., width); PyTorch's CosineEmbeddingLoss with target 1.

Note
Computed on the host in double precision; the gradient comes back on the student's device. A row whose student or teacher vector is zero counts as cos = 0, with no gradient.

Constructor & Destructor Documentation

◆ HiddenStateCosineLoss()

pulsatrix::HiddenStateCosineLoss::HiddenStateCosineLoss ( DeviceBackend *  backend)
explicit

Member Function Documentation

◆ backward()

Tensor pulsatrix::HiddenStateCosineLoss::backward ( ) const

Gradient with respect to the student's vectors, their shape.

Exceptions
std::logic_errorbefore forward().

◆ forward()

float pulsatrix::HiddenStateCosineLoss::forward ( const Tensor &  student,
const Tensor &  teacher,
const std::vector< float > &  mask = {} 
)
Parameters
maskOne value per row, 0 to leave it out (padding) and anything else to keep it; empty keeps every row.
Exceptions
std::invalid_argumentif the shapes differ, the rank is below 2, the tensors are on different devices, the mask has the wrong length, or it keeps no row.

The documentation for this class was generated from the following file: