pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::UnicodeRegex Class Reference

A compiled pattern, matched leftmost-first with backtracking as Oniguruma (which Hugging Face tokenizers uses) and Perl do. More...

#include <unicode_regex.hpp>

Classes

struct  Match
 A match: a byte range [begin, end) of the subject. More...
 

Public Member Functions

 UnicodeRegex (std::string_view pattern)
 
 ~UnicodeRegex ()
 
 UnicodeRegex (UnicodeRegex &&) noexcept
 
UnicodeRegex & operator= (UnicodeRegex &&) noexcept
 
std::vector< Match > find_all (std::string_view text) const
 Every non-overlapping match in valid UTF-8 text, scanning left to right. Empty matches are skipped.
 
const std::string & pattern () const
 The pattern as given.
 

Detailed Description

A compiled pattern, matched leftmost-first with backtracking as Oniguruma (which Hugging Face tokenizers uses) and Perl do.

Supported syntax:

  • alternation |; groups (...), (?:...), (?i:...) (case-insensitive), lookaheads (?=...) and (?!...)
  • quantifiers ?, *, +, {n}, {n,}, {n,m}, each optionally lazy (??, *?, ...)
  • . (any character but \n), literals, character classes [...] and [^...] with ranges
  • \p{X} and \P{X} for general categories (L, Lu, Ll, Lt, Lm, Lo, M, Mn, N, Nd, P, S, Z, C, ... and L&); \s (White_Space), \d (Nd), \w (letters, marks, Nd and Pc) and their negations; \r, \n, \t, \f, \v, \0, \xHH, \x{H...}, \uHHHH, and escaped punctuation

Case-insensitive matching folds ASCII letters, plus U+017F (long s) and U+212A (Kelvin sign), which is all the contraction patterns ((?i:'s|'t|...)) can meet.

Constructor & Destructor Documentation

◆ UnicodeRegex() [1/2]

pulsatrix::UnicodeRegex::UnicodeRegex ( std::string_view  pattern)
explicit
Exceptions
std::invalid_argumentfor a syntax error or unsupported syntax, naming the position.

◆ ~UnicodeRegex()

pulsatrix::UnicodeRegex::~UnicodeRegex ( )

◆ UnicodeRegex() [2/2]

pulsatrix::UnicodeRegex::UnicodeRegex ( UnicodeRegex &&  )
noexcept

Member Function Documentation

◆ find_all()

std::vector< Match > pulsatrix::UnicodeRegex::find_all ( std::string_view  text) const

Every non-overlapping match in valid UTF-8 text, scanning left to right. Empty matches are skipped.

Exceptions
std::invalid_argumentif text isn't valid UTF-8.

◆ operator=()

UnicodeRegex & pulsatrix::UnicodeRegex::operator= ( UnicodeRegex &&  )
noexcept

◆ pattern()

const std::string & pulsatrix::UnicodeRegex::pattern ( ) const
inline

The pattern as given.


The documentation for this class was generated from the following file: