Per-word scores from per-token scores (TOK-4): explanations are made per token but read per word, so relevance, attributions or probe outputs are merged into the words of the input using the tokenizer's offsets.
More...
|
| std::vector< Offset > | pulsatrix::SplitWords (std::string_view text, WordSplit split=WordSplit::WordsAndPunctuation) |
| | The words of valid UTF-8 text, as byte ranges in order. Scripts written without spaces (Chinese, Japanese, Thai) come out as long runs; pass a pattern for those.
|
| |
| std::vector< Offset > | pulsatrix::SplitWords (std::string_view text, const UnicodeRegex &pattern) |
| | The words of text: every match of pattern.
|
| |
| WordScores | pulsatrix::AggregateToWords (std::string_view text, const std::vector< Offset > &token_offsets, const std::vector< float > &token_scores, const std::vector< Offset > &words, WordAggregation aggregation=WordAggregation::Sum) |
| | Merges per-token scores into the given words.
|
| |
| WordScores | pulsatrix::AggregateToWords (std::string_view text, const Encoding &encoding, const std::vector< float > &token_scores, WordAggregation aggregation=WordAggregation::Sum, WordSplit split=WordSplit::WordsAndPunctuation) |
| | AggregateToWords over an Encoding of text, with words from SplitWords(text, split).
|
| |
Per-word scores from per-token scores (TOK-4): explanations are made per token but read per word, so relevance, attributions or probe outputs are merged into the words of the input using the tokenizer's offsets.