pulsatrix 1.1.0
Loading...
Searching...
No Matches
word_scores.hpp
Go to the documentation of this file.
1
7#pragma once
8
9#include <cstddef>
10#include <string>
11#include <string_view>
12#include <vector>
13
16
17namespace pulsatrix {
18
20enum class WordSplit {
27};
28
32[[nodiscard]] std::vector<Offset> SplitWords(std::string_view text, WordSplit split = WordSplit::WordsAndPunctuation);
34[[nodiscard]] std::vector<Offset> SplitWords(std::string_view text, const UnicodeRegex& pattern);
35
43enum class WordAggregation {
46 Sum,
48 Mean,
50 Max,
53 MaxAbs,
54};
55
57struct WordScore {
59 std::string text;
60 float score = 0.0f;
63 std::vector<size_t> tokens;
64};
65
67struct WordScores {
68 std::vector<WordScore> words;
71 float unassigned = 0.0f;
72};
73
83[[nodiscard]] WordScores AggregateToWords(std::string_view text, const std::vector<Offset>& token_offsets,
84 const std::vector<float>& token_scores, const std::vector<Offset>& words,
86
88[[nodiscard]] WordScores AggregateToWords(std::string_view text, const Encoding& encoding,
89 const std::vector<float>& token_scores,
92
93} // namespace pulsatrix
A compiled pattern, matched leftmost-first with backtracking as Oniguruma (which Hugging Face tokeniz...
Definition unicode_regex.hpp:35
Definition acquisition_functions.hpp:16
std::vector< Offset > SplitWords(std::string_view text, WordSplit split=WordSplit::WordsAndPunctuation)
The words of valid UTF-8 text, as byte ranges in order. Scripts written without spaces (Chinese,...
WordSplit
How the text is cut into words.
Definition word_scores.hpp:20
@ WordsAndPunctuation
Runs of letters, marks, digits and underscores, and runs of other non-space characters (\w+|[^\w\s]+,...
@ Whitespace
Runs of non-space characters (\S+): "don't!" is one word.
WordAggregation
How a word's score is made from the scores of the tokens that overlap it.
Definition word_scores.hpp:43
@ MaxAbs
The score with the largest magnitude among the overlapping tokens, sign kept (for signed attributions...
@ Sum
Σ share × score. Conserves the total: the words' scores plus the unassigned score add up to the token...
WordScores AggregateToWords(std::string_view text, const std::vector< Offset > &token_offsets, const std::vector< float > &token_scores, const std::vector< Offset > &words, WordAggregation aggregation=WordAggregation::Sum)
Merges per-token scores into the given words.
@ Mean
One point per sample: the mean over its positions.
@ Max
One point per sample: the maximum over its positions, per feature.
The result of encoding one text.
Definition text_tokenizer.hpp:37
A byte range [begin, end) of the text passed to TextTokenizer::encode().
Definition text_tokenizer.hpp:22
One word and its score.
Definition word_scores.hpp:57
Offset span
Definition word_scores.hpp:58
float score
Definition word_scores.hpp:60
std::string text
Definition word_scores.hpp:59
std::vector< size_t > tokens
Indices of the tokens that overlap the word, in order. Empty when none does (a character the vocabula...
Definition word_scores.hpp:63
Per-word scores for one text.
Definition word_scores.hpp:67
float unassigned
The total score of tokens that overlap no word: BOS, EOS and other tokens with an empty offset,...
Definition word_scores.hpp:71
std::vector< WordScore > words
Definition word_scores.hpp:68
The tokenizer interface (TOK-1): a pipeline of normalizer, pre-tokenizer, model, post-processor and d...
A small backtracking regex engine over Unicode code points, for the pre-tokenizer patterns in tokeniz...