|
pulsatrix 1.1.0
|
Turns token strings back into text. More...
#include <text_tokenizer.hpp>

Public Member Functions | |
| virtual | ~Decoder ()=default |
| virtual std::string | decode (const std::vector< std::string > &tokens) const |
| The decoded text: decode_chain() joined. | |
| virtual std::vector< std::string > | decode_chain (std::vector< std::string > tokens) const |
| The tokens after this decoder: by default decode()'s text as one token. | |
Turns token strings back into text.
Turns token strings back into text. As in Hugging Face, a decoder maps a token list to a token list (decode_chain), so decoders compose (SequenceDecoder), and decode() joins the result. Override at least one of the two.
|
virtualdefault |
|
virtual |
The decoded text: decode_chain() joined.
Reimplemented in pulsatrix::ByteLevelDecoder, and pulsatrix::FuseDecoder.
|
virtual |
The tokens after this decoder: by default decode()'s text as one token.
Reimplemented in pulsatrix::ReplaceDecoder, pulsatrix::ByteFallbackDecoder, pulsatrix::StripDecoder, pulsatrix::MetaspaceDecoder, and pulsatrix::SequenceDecoder.