pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::PreTokenizer Class Referenceabstract

Splits text into the pieces the model tokenizes separately, and may change them (byte- level mapping). More...

#include <text_tokenizer.hpp>

Inheritance diagram for pulsatrix::PreTokenizer:

Public Member Functions

virtual ~PreTokenizer ()=default
 
virtual void pre_tokenize (std::vector< NormalizedString > &splits) const =0
 Replaces splits with their further-split (or transformed) pieces, in order.
 

Detailed Description

Splits text into the pieces the model tokenizes separately, and may change them (byte- level mapping).

Constructor & Destructor Documentation

◆ ~PreTokenizer()

virtual pulsatrix::PreTokenizer::~PreTokenizer ( )
virtualdefault

Member Function Documentation

◆ pre_tokenize()

virtual void pulsatrix::PreTokenizer::pre_tokenize ( std::vector< NormalizedString > &  splits) const
pure virtual

The documentation for this class was generated from the following file: