sum over tokens of -log softmax(logits)[target], divided by a normalizer; tokens whose target is the ignore index (padding) contribute nothing.
More...
#include <token_cross_entropy_loss.hpp>
|
| static constexpr int64_t | kIgnoreIndex = -100 |
| | PyTorch's default ignore_index.
|
| |
sum over tokens of -log softmax(logits)[target], divided by a normalizer; tokens whose target is the ignore index (padding) contribute nothing.
- Note
- Logits are (..., classes): every row along the last dimension is one token. Targets have the leading shape (...), as whole-number floats (the library's index convention).
-
Gradient accumulation: averaging each micro-batch's own mean is wrong when micro-batches hold different numbers of real tokens (the bug Hugging Face Trainer fixed in v4.46). Count the tokens of the whole accumulation window with CountTargetTokens() and pass that total as every micro-batch's normalizer; the accumulated gradients then equal one big batch's.
-
Device-generic: the per-row softmax and gradient run through DeviceBackend::rl_rows, the same row kernels as PolicyGradientLoss with a weight of 1 per real token and 0 per ignored one. The targets are checked on their own device (HIP-6): forward() reads back four numbers, the loss, the token count and two error counts, and nothing else.
◆ TokenCrossEntropyLoss()
◆ backward()
| Tensor pulsatrix::TokenCrossEntropyLoss::backward |
( |
| ) |
const |
Gradient w.r.t. the logits, same shape.
- Exceptions
-
◆ forward()
| float pulsatrix::TokenCrossEntropyLoss::forward |
( |
const Tensor & |
logits, |
|
|
const Tensor & |
targets, |
|
|
float |
normalizer = 0.0f |
|
) |
| |
Computes the loss and caches what backward() needs.
- Parameters
-
| normalizer | What the summed loss is divided by. 0 (the default) means this batch's own number of real tokens, i.e. the mean. |
- Returns
- The loss; 0 when every token is ignored.
- Exceptions
-
| std::invalid_argument | if logits are not rank >= 2 with at least one row and class, targets don't have logits' leading shape, a target is neither a class index nor the ignore index, or the normalizer is negative or not finite. |
◆ num_tokens()
| int64_t pulsatrix::TokenCrossEntropyLoss::num_tokens |
( |
| ) |
const |
|
inline |
Real (not ignored) tokens in the last forward().
◆ kIgnoreIndex
| constexpr int64_t pulsatrix::TokenCrossEntropyLoss::kIgnoreIndex = -100 |
|
staticconstexpr |
PyTorch's default ignore_index.
The documentation for this class was generated from the following file: