|
pulsatrix 1.1.0
|
This is the complete list of members for pulsatrix::ByteLevelPreTokenizer, including all inherited members.
| ByteLevelPreTokenizer(bool add_prefix_space, bool use_regex) | pulsatrix::ByteLevelPreTokenizer | |
| pre_tokenize(std::vector< NormalizedString > &splits) const override | pulsatrix::ByteLevelPreTokenizer | virtual |
| ~PreTokenizer()=default | pulsatrix::PreTokenizer | virtual |