|
pulsatrix 1.1.0
|
Maps byte-level characters back to bytes; invalid UTF-8 becomes U+FFFD, as in Hugging Face. More...
#include <byte_level_bpe.hpp>


Public Member Functions | |
| std::string | decode (const std::vector< std::string > &tokens) const override |
| The decoded text: decode_chain() joined. | |
Public Member Functions inherited from pulsatrix::Decoder | |
| virtual | ~Decoder ()=default |
| virtual std::vector< std::string > | decode_chain (std::vector< std::string > tokens) const |
| The tokens after this decoder: by default decode()'s text as one token. | |
Maps byte-level characters back to bytes; invalid UTF-8 becomes U+FFFD, as in Hugging Face.
|
overridevirtual |
The decoded text: decode_chain() joined.
Reimplemented from pulsatrix::Decoder.