pulsatrix 1.1.0
Loading...
Searching...
No Matches
text_relevance.hpp
Go to the documentation of this file.
1
7#pragma once
8
9#include <string>
10#include <string_view>
11#include <vector>
12
16
17namespace pulsatrix {
18
31[[nodiscard]] TokenRelevanceDocument MakeTokenRelevanceDocument(std::string_view text, const Encoding& encoding,
32 const std::vector<float>& token_scores, std::string method,
33 std::string target = "", bool include_special_tokens = true);
34
40[[nodiscard]] TokenRelevanceDocument MakeWordRelevanceDocument(std::string_view text, const WordScores& words,
41 std::string method, std::string target = "");
42
43} // namespace pulsatrix
Versioned JSON documents (pulsatrix.<kind>.v1) that every renderer reads.
Definition acquisition_functions.hpp:16
TokenRelevanceDocument MakeTokenRelevanceDocument(std::string_view text, const Encoding &encoding, const std::vector< float > &token_scores, std::string method, std::string target="", bool include_special_tokens=true)
A token-level document for text from its Encoding and one score per token.
TokenRelevanceDocument MakeWordRelevanceDocument(std::string_view text, const WordScores &words, std::string method, std::string target="")
A word-level document for text from AggregateToWords(): each word scored, the text between words as u...
The result of encoding one text.
Definition text_tokenizer.hpp:37
pulsatrix.token_relevance.v1: relevance scores over the pieces of a text, in reading order....
Definition document.hpp:104
Per-word scores for one text.
Definition word_scores.hpp:67
The tokenizer interface (TOK-1): a pipeline of normalizer, pre-tokenizer, model, post-processor and d...
Per-word scores from per-token scores (TOK-4): explanations are made per token but read per word,...