|
pulsatrix 1.1.0
|
A small backtracking regex engine over Unicode code points, for the pre-tokenizer patterns in tokenizer.json files (TOK-2). std::regex can't match Unicode categories.
More...
#include <cstddef>#include <memory>#include <optional>#include <string>#include <string_view>#include <vector>

Go to the source code of this file.
Classes | |
| class | pulsatrix::UnicodeRegex |
A compiled pattern, matched leftmost-first with backtracking as Oniguruma (which Hugging Face tokenizers uses) and Perl do. More... | |
| struct | pulsatrix::UnicodeRegex::Match |
A match: a byte range [begin, end) of the subject. More... | |
Namespaces | |
| namespace | pulsatrix |
A small backtracking regex engine over Unicode code points, for the pre-tokenizer patterns in tokenizer.json files (TOK-2). std::regex can't match Unicode categories.