pulsatrix 1.1.0
Loading...
Searching...
No Matches
unicode_regex.hpp File Reference

A small backtracking regex engine over Unicode code points, for the pre-tokenizer patterns in tokenizer.json files (TOK-2). std::regex can't match Unicode categories. More...

#include <cstddef>
#include <memory>
#include <optional>
#include <string>
#include <string_view>
#include <vector>
Include dependency graph for unicode_regex.hpp:
This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes

class  pulsatrix::UnicodeRegex
 A compiled pattern, matched leftmost-first with backtracking as Oniguruma (which Hugging Face tokenizers uses) and Perl do. More...
 
struct  pulsatrix::UnicodeRegex::Match
 A match: a byte range [begin, end) of the subject. More...
 

Namespaces

namespace  pulsatrix
 

Detailed Description

A small backtracking regex engine over Unicode code points, for the pre-tokenizer patterns in tokenizer.json files (TOK-2). std::regex can't match Unicode categories.