|
pulsatrix 1.1.0
|
Spaces become a replacement character (▁), one is put in front of each piece (or of the text's first piece only), and pieces are optionally split before each one (Hugging Face's Metaspace pre-tokenizer).
More...
#include <sentencepiece_bpe.hpp>


Public Member Functions | |
| MetaspacePreTokenizer (std::string replacement, PrependScheme scheme, bool split) | |
| void | pre_tokenize (std::vector< NormalizedString > &splits) const override |
Replaces splits with their further-split (or transformed) pieces, in order. | |
Public Member Functions inherited from pulsatrix::PreTokenizer | |
| virtual | ~PreTokenizer ()=default |
Spaces become a replacement character (▁), one is put in front of each piece (or of the text's first piece only), and pieces are optionally split before each one (Hugging Face's Metaspace pre-tokenizer).
| pulsatrix::MetaspacePreTokenizer::MetaspacePreTokenizer | ( | std::string | replacement, |
| PrependScheme | scheme, | ||
| bool | split | ||
| ) |
|
overridevirtual |
Replaces splits with their further-split (or transformed) pieces, in order.
Implements pulsatrix::PreTokenizer.