TokenRelevanceDocument MakeTokenRelevanceDocument(std::string_view text, const Encoding &encoding, const std::vector< float > &token_scores, std::string method, std::string target="", bool include_special_tokens=true)
A token-level document for text from its Encoding and one score per token.
TokenRelevanceDocument MakeWordRelevanceDocument(std::string_view text, const WordScores &words, std::string method, std::string target="")
A word-level document for text from AggregateToWords(): each word scored, the text between words as u...
The tokenizer interface (TOK-1): a pipeline of normalizer, pre-tokenizer, model, post-processor and d...