A compiled pattern, matched leftmost-first with backtracking as Oniguruma (which Hugging Face tokenizers uses) and Perl do.
More...
#include <unicode_regex.hpp>
|
| struct | Match |
| | A match: a byte range [begin, end) of the subject. More...
|
| |
A compiled pattern, matched leftmost-first with backtracking as Oniguruma (which Hugging Face tokenizers uses) and Perl do.
Supported syntax:
- alternation
|; groups (...), (?:...), (?i:...) (case-insensitive), lookaheads (?=...) and (?!...)
- quantifiers
?, *, +, {n}, {n,}, {n,m}, each optionally lazy (??, *?, ...)
. (any character but \n), literals, character classes [...] and [^...] with ranges
\p{X} and \P{X} for general categories (L, Lu, Ll, Lt, Lm, Lo, M, Mn, N, Nd, P, S, Z, C, ... and L&); \s (White_Space), \d (Nd), \w (letters, marks, Nd and Pc) and their negations; \r, \n, \t, \f, \v, \0, \xHH, \x{H...}, \uHHHH, and escaped punctuation
Case-insensitive matching folds ASCII letters, plus U+017F (long s) and U+212A (Kelvin sign), which is all the contraction patterns ((?i:'s|'t|...)) can meet.
◆ UnicodeRegex() [1/2]
| pulsatrix::UnicodeRegex::UnicodeRegex |
( |
std::string_view |
pattern | ) |
|
|
explicit |
- Exceptions
-
| std::invalid_argument | for a syntax error or unsupported syntax, naming the position. |
◆ ~UnicodeRegex()
| pulsatrix::UnicodeRegex::~UnicodeRegex |
( |
| ) |
|
◆ UnicodeRegex() [2/2]
◆ find_all()
| std::vector< Match > pulsatrix::UnicodeRegex::find_all |
( |
std::string_view |
text | ) |
const |
Every non-overlapping match in valid UTF-8 text, scanning left to right. Empty matches are skipped.
- Exceptions
-
| std::invalid_argument | if text isn't valid UTF-8. |
◆ operator=()
◆ pattern()
| const std::string & pulsatrix::UnicodeRegex::pattern |
( |
| ) |
const |
|
inline |
The documentation for this class was generated from the following file: