|
pulsatrix 1.1.0
|
Runs pre-tokenizers in order. More...
#include <byte_level_bpe.hpp>


Public Member Functions | |
| SequencePreTokenizer (std::vector< std::shared_ptr< const PreTokenizer > > pre_tokenizers) | |
| void | pre_tokenize (std::vector< NormalizedString > &splits) const override |
Replaces splits with their further-split (or transformed) pieces, in order. | |
Public Member Functions inherited from pulsatrix::PreTokenizer | |
| virtual | ~PreTokenizer ()=default |
Runs pre-tokenizers in order.
|
explicit |
|
overridevirtual |
Replaces splits with their further-split (or transformed) pieces, in order.
Implements pulsatrix::PreTokenizer.