|
pulsatrix 1.1.0
|
Turns runs of <0x41>-style byte tokens back into text; a run that isn't valid UTF-8 becomes one U+FFFD per byte (Hugging Face's ByteFallback decoder).
More...
#include <sentencepiece_bpe.hpp>


Public Member Functions | |
| std::vector< std::string > | decode_chain (std::vector< std::string > tokens) const override |
| The tokens after this decoder: by default decode()'s text as one token. | |
Public Member Functions inherited from pulsatrix::Decoder | |
| virtual | ~Decoder ()=default |
| virtual std::string | decode (const std::vector< std::string > &tokens) const |
| The decoded text: decode_chain() joined. | |
Turns runs of <0x41>-style byte tokens back into text; a run that isn't valid UTF-8 becomes one U+FFFD per byte (Hugging Face's ByteFallback decoder).
|
overridevirtual |
The tokens after this decoder: by default decode()'s text as one token.
Reimplemented from pulsatrix::Decoder.