pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::SequencePreTokenizer Class Reference

Runs pre-tokenizers in order. More...

#include <byte_level_bpe.hpp>

Inheritance diagram for pulsatrix::SequencePreTokenizer:
Collaboration diagram for pulsatrix::SequencePreTokenizer:

Public Member Functions

 SequencePreTokenizer (std::vector< std::shared_ptr< const PreTokenizer > > pre_tokenizers)
 
void pre_tokenize (std::vector< NormalizedString > &splits) const override
 Replaces splits with their further-split (or transformed) pieces, in order.
 
- Public Member Functions inherited from pulsatrix::PreTokenizer
virtual ~PreTokenizer ()=default
 

Detailed Description

Runs pre-tokenizers in order.

Constructor & Destructor Documentation

◆ SequencePreTokenizer()

pulsatrix::SequencePreTokenizer::SequencePreTokenizer ( std::vector< std::shared_ptr< const PreTokenizer > >  pre_tokenizers)
explicit

Member Function Documentation

◆ pre_tokenize()

void pulsatrix::SequencePreTokenizer::pre_tokenize ( std::vector< NormalizedString > &  splits) const
overridevirtual

Replaces splits with their further-split (or transformed) pieces, in order.

Implements pulsatrix::PreTokenizer.


The documentation for this class was generated from the following file: