pulsatrix 1.1.0
Loading...
Searching...
No Matches
text_relevance.hpp File Reference

Token relevance documents from real explanations (VIZ-6a): a tokenizer's Encoding or TOK-4's word scores, turned into a pulsatrix.token_relevance.v1 document whose pieces read as the original text. More...

#include <string>
#include <string_view>
#include <vector>
#include "pulsatrix/text_tokenizer.hpp"
#include "pulsatrix/viz/document.hpp"
#include "pulsatrix/word_scores.hpp"
Include dependency graph for text_relevance.hpp:

Go to the source code of this file.

Namespaces

namespace  pulsatrix
 

Functions

TokenRelevanceDocument pulsatrix::MakeTokenRelevanceDocument (std::string_view text, const Encoding &encoding, const std::vector< float > &token_scores, std::string method, std::string target="", bool include_special_tokens=true)
 A token-level document for text from its Encoding and one score per token.
 
TokenRelevanceDocument pulsatrix::MakeWordRelevanceDocument (std::string_view text, const WordScores &words, std::string method, std::string target="")
 A word-level document for text from AggregateToWords(): each word scored, the text between words as unscored pieces, and the words' unassigned relevance carried over.
 

Detailed Description

Token relevance documents from real explanations (VIZ-6a): a tokenizer's Encoding or TOK-4's word scores, turned into a pulsatrix.token_relevance.v1 document whose pieces read as the original text.