pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::ByteLevelPreTokenizer Class Reference

Byte-level pre-tokenization (Hugging Face's ByteLevel): optionally prepends a space, optionally splits with the GPT-2 regex, then maps every byte to a printable character with GPT-2's table, so BPE runs over characters that stand for bytes. More...

#include <byte_level_bpe.hpp>

Inheritance diagram for pulsatrix::ByteLevelPreTokenizer:
Collaboration diagram for pulsatrix::ByteLevelPreTokenizer:

Public Member Functions

 ByteLevelPreTokenizer (bool add_prefix_space, bool use_regex)
 
void pre_tokenize (std::vector< NormalizedString > &splits) const override
 Replaces splits with their further-split (or transformed) pieces, in order.
 
- Public Member Functions inherited from pulsatrix::PreTokenizer
virtual ~PreTokenizer ()=default
 

Detailed Description

Byte-level pre-tokenization (Hugging Face's ByteLevel): optionally prepends a space, optionally splits with the GPT-2 regex, then maps every byte to a printable character with GPT-2's table, so BPE runs over characters that stand for bytes.

Constructor & Destructor Documentation

◆ ByteLevelPreTokenizer()

pulsatrix::ByteLevelPreTokenizer::ByteLevelPreTokenizer ( bool  add_prefix_space,
bool  use_regex 
)

Member Function Documentation

◆ pre_tokenize()

void pulsatrix::ByteLevelPreTokenizer::pre_tokenize ( std::vector< NormalizedString > &  splits) const
overridevirtual

Replaces splits with their further-split (or transformed) pieces, in order.

Implements pulsatrix::PreTokenizer.


The documentation for this class was generated from the following file: