|
pulsatrix 1.1.0
|
Splits on a regex or a literal string (Hugging Face's Split pre-tokenizer). More...
#include <byte_level_bpe.hpp>


Public Member Functions | |
| SplitPreTokenizer (const std::string ®ex, SplitBehavior behavior, bool invert=false) | |
| A regex pattern. | |
| void | pre_tokenize (std::vector< NormalizedString > &splits) const override |
Replaces splits with their further-split (or transformed) pieces, in order. | |
Public Member Functions inherited from pulsatrix::PreTokenizer | |
| virtual | ~PreTokenizer ()=default |
Static Public Member Functions | |
| static SplitPreTokenizer | Literal (const std::string &text, SplitBehavior behavior, bool invert=false) |
| A literal string pattern. | |
Splits on a regex or a literal string (Hugging Face's Split pre-tokenizer).
| pulsatrix::SplitPreTokenizer::SplitPreTokenizer | ( | const std::string & | regex, |
| SplitBehavior | behavior, | ||
| bool | invert = false |
||
| ) |
A regex pattern.
| std::invalid_argument | as UnicodeRegex does. |
|
static |
A literal string pattern.
|
overridevirtual |
Replaces splits with their further-split (or transformed) pieces, in order.
Implements pulsatrix::PreTokenizer.