|
pulsatrix 1.1.0
|
Splits off numeric characters (Unicode N), one by one or in runs (Hugging Face's Digits). More...
#include <byte_level_bpe.hpp>


Public Member Functions | |
| DigitsPreTokenizer (bool individual_digits) | |
| void | pre_tokenize (std::vector< NormalizedString > &splits) const override |
Replaces splits with their further-split (or transformed) pieces, in order. | |
Public Member Functions inherited from pulsatrix::PreTokenizer | |
| virtual | ~PreTokenizer ()=default |
Splits off numeric characters (Unicode N), one by one or in runs (Hugging Face's Digits).
|
explicit |
|
overridevirtual |
Replaces splits with their further-split (or transformed) pieces, in order.
Implements pulsatrix::PreTokenizer.