pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::DigitsPreTokenizer Class Reference

Splits off numeric characters (Unicode N), one by one or in runs (Hugging Face's Digits). More...

#include <byte_level_bpe.hpp>

Inheritance diagram for pulsatrix::DigitsPreTokenizer:
Collaboration diagram for pulsatrix::DigitsPreTokenizer:

Public Member Functions

 DigitsPreTokenizer (bool individual_digits)
 
void pre_tokenize (std::vector< NormalizedString > &splits) const override
 Replaces splits with their further-split (or transformed) pieces, in order.
 
- Public Member Functions inherited from pulsatrix::PreTokenizer
virtual ~PreTokenizer ()=default
 

Detailed Description

Splits off numeric characters (Unicode N), one by one or in runs (Hugging Face's Digits).

Constructor & Destructor Documentation

◆ DigitsPreTokenizer()

pulsatrix::DigitsPreTokenizer::DigitsPreTokenizer ( bool  individual_digits)
explicit

Member Function Documentation

◆ pre_tokenize()

void pulsatrix::DigitsPreTokenizer::pre_tokenize ( std::vector< NormalizedString > &  splits) const
overridevirtual

Replaces splits with their further-split (or transformed) pieces, in order.

Implements pulsatrix::PreTokenizer.


The documentation for this class was generated from the following file: