pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::SplitPreTokenizer Class Reference

Splits on a regex or a literal string (Hugging Face's Split pre-tokenizer). More...

#include <byte_level_bpe.hpp>

Inheritance diagram for pulsatrix::SplitPreTokenizer:
Collaboration diagram for pulsatrix::SplitPreTokenizer:

Public Member Functions

 SplitPreTokenizer (const std::string &regex, SplitBehavior behavior, bool invert=false)
 A regex pattern.
 
void pre_tokenize (std::vector< NormalizedString > &splits) const override
 Replaces splits with their further-split (or transformed) pieces, in order.
 
- Public Member Functions inherited from pulsatrix::PreTokenizer
virtual ~PreTokenizer ()=default
 

Static Public Member Functions

static SplitPreTokenizer Literal (const std::string &text, SplitBehavior behavior, bool invert=false)
 A literal string pattern.
 

Detailed Description

Splits on a regex or a literal string (Hugging Face's Split pre-tokenizer).

Constructor & Destructor Documentation

◆ SplitPreTokenizer()

pulsatrix::SplitPreTokenizer::SplitPreTokenizer ( const std::string &  regex,
SplitBehavior  behavior,
bool  invert = false 
)

A regex pattern.

Exceptions
std::invalid_argumentas UnicodeRegex does.

Member Function Documentation

◆ Literal()

static SplitPreTokenizer pulsatrix::SplitPreTokenizer::Literal ( const std::string &  text,
SplitBehavior  behavior,
bool  invert = false 
)
static

A literal string pattern.

◆ pre_tokenize()

void pulsatrix::SplitPreTokenizer::pre_tokenize ( std::vector< NormalizedString > &  splits) const
overridevirtual

Replaces splits with their further-split (or transformed) pieces, in order.

Implements pulsatrix::PreTokenizer.


The documentation for this class was generated from the following file: