pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::TokenRelevanceDocument Struct Reference

pulsatrix.token_relevance.v1: relevance scores over the pieces of a text, in reading order. MakeTokenRelevanceDocument() and MakeWordRelevanceDocument() (viz/text_relevance.hpp) build one from a tokenizer's Encoding or from word scores. More...

#include <document.hpp>

Public Member Functions

bool is_scored (size_t i) const
 Whether piece i carries a score.
 

Public Attributes

std::string method
 
std::vector< std::string > tokens
 The pieces as text, in order; concatenated, they read as the text. Must be valid UTF-8: decode byte-level tokens first.
 
std::vector< float > relevance
 One score per piece.
 
std::string target
 What was explained, for example the predicted next token. Optional.
 
std::string granularity = "token"
 "token" or "word": what the scored pieces are. Written only when "word".
 
std::vector< bool > scored
 Whether each piece carries a score; empty means all do. Unscored pieces are context, such as the spaces between words, drawn without a color. Written only when some piece is unscored.
 
std::optional< float > unassigned
 Relevance that belongs to no piece shown, such as a BOS token left out or tokens between words. Written only when set.
 

Detailed Description

pulsatrix.token_relevance.v1: relevance scores over the pieces of a text, in reading order. MakeTokenRelevanceDocument() and MakeWordRelevanceDocument() (viz/text_relevance.hpp) build one from a tokenizer's Encoding or from word scores.

Member Function Documentation

◆ is_scored()

bool pulsatrix::TokenRelevanceDocument::is_scored ( size_t  i) const
inline

Whether piece i carries a score.

Member Data Documentation

◆ granularity

std::string pulsatrix::TokenRelevanceDocument::granularity = "token"

"token" or "word": what the scored pieces are. Written only when "word".

◆ method

std::string pulsatrix::TokenRelevanceDocument::method

◆ relevance

std::vector<float> pulsatrix::TokenRelevanceDocument::relevance

One score per piece.

◆ scored

std::vector<bool> pulsatrix::TokenRelevanceDocument::scored

Whether each piece carries a score; empty means all do. Unscored pieces are context, such as the spaces between words, drawn without a color. Written only when some piece is unscored.

◆ target

std::string pulsatrix::TokenRelevanceDocument::target

What was explained, for example the predicted next token. Optional.

◆ tokens

std::vector<std::string> pulsatrix::TokenRelevanceDocument::tokens

The pieces as text, in order; concatenated, they read as the text. Must be valid UTF-8: decode byte-level tokens first.

◆ unassigned

std::optional<float> pulsatrix::TokenRelevanceDocument::unassigned

Relevance that belongs to no piece shown, such as a BOS token left out or tokens between words. Written only when set.


The documentation for this struct was generated from the following file: