pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::MetaspacePreTokenizer Class Reference

Spaces become a replacement character (▁), one is put in front of each piece (or of the text's first piece only), and pieces are optionally split before each one (Hugging Face's Metaspace pre-tokenizer). More...

#include <sentencepiece_bpe.hpp>

Inheritance diagram for pulsatrix::MetaspacePreTokenizer:
Collaboration diagram for pulsatrix::MetaspacePreTokenizer:

Public Member Functions

 MetaspacePreTokenizer (std::string replacement, PrependScheme scheme, bool split)
 
void pre_tokenize (std::vector< NormalizedString > &splits) const override
 Replaces splits with their further-split (or transformed) pieces, in order.
 
- Public Member Functions inherited from pulsatrix::PreTokenizer
virtual ~PreTokenizer ()=default
 

Detailed Description

Spaces become a replacement character (▁), one is put in front of each piece (or of the text's first piece only), and pieces are optionally split before each one (Hugging Face's Metaspace pre-tokenizer).

Constructor & Destructor Documentation

◆ MetaspacePreTokenizer()

pulsatrix::MetaspacePreTokenizer::MetaspacePreTokenizer ( std::string  replacement,
PrependScheme  scheme,
bool  split 
)

Member Function Documentation

◆ pre_tokenize()

void pulsatrix::MetaspacePreTokenizer::pre_tokenize ( std::vector< NormalizedString > &  splits) const
overridevirtual

Replaces splits with their further-split (or transformed) pieces, in order.

Implements pulsatrix::PreTokenizer.


The documentation for this class was generated from the following file: