84 const std::vector<float>& token_scores,
const std::vector<Offset>& words,
89 const std::vector<float>& token_scores,
A compiled pattern, matched leftmost-first with backtracking as Oniguruma (which Hugging Face tokeniz...
Definition unicode_regex.hpp:35
Definition acquisition_functions.hpp:16
std::vector< Offset > SplitWords(std::string_view text, WordSplit split=WordSplit::WordsAndPunctuation)
The words of valid UTF-8 text, as byte ranges in order. Scripts written without spaces (Chinese,...
WordSplit
How the text is cut into words.
Definition word_scores.hpp:20
@ WordsAndPunctuation
Runs of letters, marks, digits and underscores, and runs of other non-space characters (\w+|[^\w\s]+,...
@ Whitespace
Runs of non-space characters (\S+): "don't!" is one word.
WordAggregation
How a word's score is made from the scores of the tokens that overlap it.
Definition word_scores.hpp:43
@ MaxAbs
The score with the largest magnitude among the overlapping tokens, sign kept (for signed attributions...
@ Sum
Σ share × score. Conserves the total: the words' scores plus the unassigned score add up to the token...
WordScores AggregateToWords(std::string_view text, const std::vector< Offset > &token_offsets, const std::vector< float > &token_scores, const std::vector< Offset > &words, WordAggregation aggregation=WordAggregation::Sum)
Merges per-token scores into the given words.
@ Mean
One point per sample: the mean over its positions.
@ Max
One point per sample: the maximum over its positions, per feature.
The result of encoding one text.
Definition text_tokenizer.hpp:37
A byte range [begin, end) of the text passed to TextTokenizer::encode().
Definition text_tokenizer.hpp:22
One word and its score.
Definition word_scores.hpp:57
Offset span
Definition word_scores.hpp:58
float score
Definition word_scores.hpp:60
std::string text
Definition word_scores.hpp:59
std::vector< size_t > tokens
Indices of the tokens that overlap the word, in order. Empty when none does (a character the vocabula...
Definition word_scores.hpp:63
Per-word scores for one text.
Definition word_scores.hpp:67
float unassigned
The total score of tokens that overlap no word: BOS, EOS and other tokens with an empty offset,...
Definition word_scores.hpp:71
std::vector< WordScore > words
Definition word_scores.hpp:68
The tokenizer interface (TOK-1): a pipeline of normalizer, pre-tokenizer, model, post-processor and d...
A small backtracking regex engine over Unicode code points, for the pre-tokenizer patterns in tokeniz...