|
pulsatrix 1.1.0
|
The replacement character back to spaces (Hugging Face's Metaspace decoder). Unless the scheme is Never, every replacement in the first token is dropped, as Hugging Face does, not only the one the pre-tokenizer put in front. More...
#include <sentencepiece_bpe.hpp>


Public Member Functions | |
| MetaspaceDecoder (std::string replacement, PrependScheme scheme) | |
| std::vector< std::string > | decode_chain (std::vector< std::string > tokens) const override |
| The tokens after this decoder: by default decode()'s text as one token. | |
Public Member Functions inherited from pulsatrix::Decoder | |
| virtual | ~Decoder ()=default |
| virtual std::string | decode (const std::vector< std::string > &tokens) const |
| The decoded text: decode_chain() joined. | |
The replacement character back to spaces (Hugging Face's Metaspace decoder). Unless the scheme is Never, every replacement in the first token is dropped, as Hugging Face does, not only the one the pre-tokenizer put in front.
| pulsatrix::MetaspaceDecoder::MetaspaceDecoder | ( | std::string | replacement, |
| PrependScheme | scheme | ||
| ) |
|
overridevirtual |
The tokens after this decoder: by default decode()'s text as one token.
Reimplemented from pulsatrix::Decoder.