pulsatrix 1.1.0
Loading...
Searching...
No Matches
word_scores.hpp File Reference

Per-word scores from per-token scores (TOK-4): explanations are made per token but read per word, so relevance, attributions or probe outputs are merged into the words of the input using the tokenizer's offsets. More...

#include <cstddef>
#include <string>
#include <string_view>
#include <vector>
#include "pulsatrix/text_tokenizer.hpp"
#include "pulsatrix/unicode_regex.hpp"
Include dependency graph for word_scores.hpp:
This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes

struct  pulsatrix::WordScore
 One word and its score. More...
 
struct  pulsatrix::WordScores
 Per-word scores for one text. More...
 

Namespaces

namespace  pulsatrix
 

Enumerations

enum class  pulsatrix::WordSplit { pulsatrix::WordsAndPunctuation , pulsatrix::Whitespace }
 How the text is cut into words. More...
 
enum class  pulsatrix::WordAggregation { pulsatrix::Sum , pulsatrix::Mean , pulsatrix::Max , pulsatrix::MaxAbs }
 How a word's score is made from the scores of the tokens that overlap it. More...
 

Functions

std::vector< Offset > pulsatrix::SplitWords (std::string_view text, WordSplit split=WordSplit::WordsAndPunctuation)
 The words of valid UTF-8 text, as byte ranges in order. Scripts written without spaces (Chinese, Japanese, Thai) come out as long runs; pass a pattern for those.
 
std::vector< Offset > pulsatrix::SplitWords (std::string_view text, const UnicodeRegex &pattern)
 The words of text: every match of pattern.
 
WordScores pulsatrix::AggregateToWords (std::string_view text, const std::vector< Offset > &token_offsets, const std::vector< float > &token_scores, const std::vector< Offset > &words, WordAggregation aggregation=WordAggregation::Sum)
 Merges per-token scores into the given words.
 
WordScores pulsatrix::AggregateToWords (std::string_view text, const Encoding &encoding, const std::vector< float > &token_scores, WordAggregation aggregation=WordAggregation::Sum, WordSplit split=WordSplit::WordsAndPunctuation)
 AggregateToWords over an Encoding of text, with words from SplitWords(text, split).
 

Detailed Description

Per-word scores from per-token scores (TOK-4): explanations are made per token but read per word, so relevance, attributions or probe outputs are merged into the words of the input using the tokenizer's offsets.