|
pulsatrix 1.1.0
|
Splits text into the pieces the model tokenizes separately, and may change them (byte- level mapping). More...
#include <text_tokenizer.hpp>

Public Member Functions | |
| virtual | ~PreTokenizer ()=default |
| virtual void | pre_tokenize (std::vector< NormalizedString > &splits) const =0 |
Replaces splits with their further-split (or transformed) pieces, in order. | |
Splits text into the pieces the model tokenizes separately, and may change them (byte- level mapping).
|
virtualdefault |
|
pure virtual |
Replaces splits with their further-split (or transformed) pieces, in order.
Implemented in pulsatrix::SplitPreTokenizer, pulsatrix::DigitsPreTokenizer, pulsatrix::ByteLevelPreTokenizer, pulsatrix::SequencePreTokenizer, pulsatrix::MetaspacePreTokenizer, and pulsatrix::WordPunctuationSplit.