|
pulsatrix 1.1.0
|
Token relevance documents from real explanations (VIZ-6a): a tokenizer's Encoding or TOK-4's word scores, turned into a pulsatrix.token_relevance.v1 document whose pieces read as the original text.
More...
#include <string>#include <string_view>#include <vector>#include "pulsatrix/text_tokenizer.hpp"#include "pulsatrix/viz/document.hpp"#include "pulsatrix/word_scores.hpp"
Go to the source code of this file.
Namespaces | |
| namespace | pulsatrix |
Functions | |
| TokenRelevanceDocument | pulsatrix::MakeTokenRelevanceDocument (std::string_view text, const Encoding &encoding, const std::vector< float > &token_scores, std::string method, std::string target="", bool include_special_tokens=true) |
A token-level document for text from its Encoding and one score per token. | |
| TokenRelevanceDocument | pulsatrix::MakeWordRelevanceDocument (std::string_view text, const WordScores &words, std::string method, std::string target="") |
A word-level document for text from AggregateToWords(): each word scored, the text between words as unscored pieces, and the words' unassigned relevance carried over. | |
Token relevance documents from real explanations (VIZ-6a): a tokenizer's Encoding or TOK-4's word scores, turned into a pulsatrix.token_relevance.v1 document whose pieces read as the original text.