pulsatrix.token_relevance.v1: relevance scores over the pieces of a text, in reading order. MakeTokenRelevanceDocument() and MakeWordRelevanceDocument() (viz/text_relevance.hpp) build one from a tokenizer's Encoding or from word scores.
More...
#include <document.hpp>
|
| bool | is_scored (size_t i) const |
| | Whether piece i carries a score.
|
| |
|
| std::string | method |
| |
| std::vector< std::string > | tokens |
| | The pieces as text, in order; concatenated, they read as the text. Must be valid UTF-8: decode byte-level tokens first.
|
| |
| std::vector< float > | relevance |
| | One score per piece.
|
| |
| std::string | target |
| | What was explained, for example the predicted next token. Optional.
|
| |
| std::string | granularity = "token" |
| | "token" or "word": what the scored pieces are. Written only when "word".
|
| |
| std::vector< bool > | scored |
| | Whether each piece carries a score; empty means all do. Unscored pieces are context, such as the spaces between words, drawn without a color. Written only when some piece is unscored.
|
| |
| std::optional< float > | unassigned |
| | Relevance that belongs to no piece shown, such as a BOS token left out or tokens between words. Written only when set.
|
| |
pulsatrix.token_relevance.v1: relevance scores over the pieces of a text, in reading order. MakeTokenRelevanceDocument() and MakeWordRelevanceDocument() (viz/text_relevance.hpp) build one from a tokenizer's Encoding or from word scores.
◆ is_scored()
| bool pulsatrix::TokenRelevanceDocument::is_scored |
( |
size_t |
i | ) |
const |
|
inline |
Whether piece i carries a score.
◆ granularity
| std::string pulsatrix::TokenRelevanceDocument::granularity = "token" |
"token" or "word": what the scored pieces are. Written only when "word".
◆ method
| std::string pulsatrix::TokenRelevanceDocument::method |
◆ relevance
| std::vector<float> pulsatrix::TokenRelevanceDocument::relevance |
◆ scored
| std::vector<bool> pulsatrix::TokenRelevanceDocument::scored |
Whether each piece carries a score; empty means all do. Unscored pieces are context, such as the spaces between words, drawn without a color. Written only when some piece is unscored.
◆ target
| std::string pulsatrix::TokenRelevanceDocument::target |
What was explained, for example the predicted next token. Optional.
◆ tokens
| std::vector<std::string> pulsatrix::TokenRelevanceDocument::tokens |
The pieces as text, in order; concatenated, they read as the text. Must be valid UTF-8: decode byte-level tokens first.
◆ unassigned
| std::optional<float> pulsatrix::TokenRelevanceDocument::unassigned |
Relevance that belongs to no piece shown, such as a BOS token left out or tokens between words. Written only when set.
The documentation for this struct was generated from the following file: