pulsatrix 1.1.0
Loading...
Searching...
No Matches
token_cross_entropy_loss.hpp File Reference

Cross-entropy over a batch of tokens, with padding ignored and a caller-chosen normalizer for exact gradient accumulation (roadmap TRN-5). More...

#include <cstdint>
#include "pulsatrix/tensor.hpp"
Include dependency graph for token_cross_entropy_loss.hpp:
This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes

class  pulsatrix::TokenCrossEntropyLoss
 sum over tokens of -log softmax(logits)[target], divided by a normalizer; tokens whose target is the ignore index (padding) contribute nothing. More...
 

Namespaces

namespace  pulsatrix
 

Functions

int64_t pulsatrix::CountTargetTokens (const Tensor &targets, int64_t ignore_index=TokenCrossEntropyLoss::kIgnoreIndex)
 Number of entries of targets that aren't ignore_index: the normalizer for one accumulation window when summed over its micro-batches' targets.
 

Detailed Description

Cross-entropy over a batch of tokens, with padding ignored and a caller-chosen normalizer for exact gradient accumulation (roadmap TRN-5).