|
pulsatrix 1.1.0
|
Removes up to start leading and stop trailing copies of a character from every token (Hugging Face's Strip decoder).
More...
#include <sentencepiece_bpe.hpp>


Public Member Functions | |
| StripDecoder (std::string content, size_t start, size_t stop) | |
| std::vector< std::string > | decode_chain (std::vector< std::string > tokens) const override |
| The tokens after this decoder: by default decode()'s text as one token. | |
Public Member Functions inherited from pulsatrix::Decoder | |
| virtual | ~Decoder ()=default |
| virtual std::string | decode (const std::vector< std::string > &tokens) const |
| The decoded text: decode_chain() joined. | |
Removes up to start leading and stop trailing copies of a character from every token (Hugging Face's Strip decoder).
| pulsatrix::StripDecoder::StripDecoder | ( | std::string | content, |
| size_t | start, | ||
| size_t | stop | ||
| ) |
|
overridevirtual |
The tokens after this decoder: by default decode()'s text as one token.
Reimplemented from pulsatrix::Decoder.