pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::TokenCrossEntropyLoss Class Reference

sum over tokens of -log softmax(logits)[target], divided by a normalizer; tokens whose target is the ignore index (padding) contribute nothing. More...

#include <token_cross_entropy_loss.hpp>

Public Member Functions

 TokenCrossEntropyLoss (DeviceBackend *backend, int64_t ignore_index=kIgnoreIndex)
 
float forward (const Tensor &logits, const Tensor &targets, float normalizer=0.0f)
 Computes the loss and caches what backward() needs.
 
Tensor backward () const
 Gradient w.r.t. the logits, same shape.
 
int64_t num_tokens () const
 Real (not ignored) tokens in the last forward().
 

Static Public Attributes

static constexpr int64_t kIgnoreIndex = -100
 PyTorch's default ignore_index.
 

Detailed Description

sum over tokens of -log softmax(logits)[target], divided by a normalizer; tokens whose target is the ignore index (padding) contribute nothing.

Note
Logits are (..., classes): every row along the last dimension is one token. Targets have the leading shape (...), as whole-number floats (the library's index convention).
Gradient accumulation: averaging each micro-batch's own mean is wrong when micro-batches hold different numbers of real tokens (the bug Hugging Face Trainer fixed in v4.46). Count the tokens of the whole accumulation window with CountTargetTokens() and pass that total as every micro-batch's normalizer; the accumulated gradients then equal one big batch's.
Device-generic: the per-row softmax and gradient run through DeviceBackend::rl_rows, the same row kernels as PolicyGradientLoss with a weight of 1 per real token and 0 per ignored one. The targets are checked on their own device (HIP-6): forward() reads back four numbers, the loss, the token count and two error counts, and nothing else.

Constructor & Destructor Documentation

◆ TokenCrossEntropyLoss()

pulsatrix::TokenCrossEntropyLoss::TokenCrossEntropyLoss ( DeviceBackend *  backend,
int64_t  ignore_index = kIgnoreIndex 
)
explicit

Member Function Documentation

◆ backward()

Tensor pulsatrix::TokenCrossEntropyLoss::backward ( ) const

Gradient w.r.t. the logits, same shape.

Exceptions
std::logic_errorbefore forward().

◆ forward()

float pulsatrix::TokenCrossEntropyLoss::forward ( const Tensor &  logits,
const Tensor &  targets,
float  normalizer = 0.0f 
)

Computes the loss and caches what backward() needs.

Parameters
normalizerWhat the summed loss is divided by. 0 (the default) means this batch's own number of real tokens, i.e. the mean.
Returns
The loss; 0 when every token is ignored.
Exceptions
std::invalid_argumentif logits are not rank >= 2 with at least one row and class, targets don't have logits' leading shape, a target is neither a class index nor the ignore index, or the normalizer is negative or not finite.

◆ num_tokens()

int64_t pulsatrix::TokenCrossEntropyLoss::num_tokens ( ) const
inline

Real (not ignored) tokens in the last forward().

Member Data Documentation

◆ kIgnoreIndex

constexpr int64_t pulsatrix::TokenCrossEntropyLoss::kIgnoreIndex = -100
staticconstexpr

PyTorch's default ignore_index.


The documentation for this class was generated from the following file: