pulsatrix 1.1.0
Loading...
Searching...
No Matches
sentencepiece_bpe.hpp File Reference

The components SentencePiece-style BPE tokenizers are built from (TOK-3): spaces as ▁ (Replace, Prepend or Metaspace), byte fallback in BpeModel (BpeOptions::byte_fallback), and the decoder chain that undoes them, each following Hugging Face tokenizers. Gemma 3, Llama 2, Mistral and TinyLlama tokenize this way. More...

#include <memory>
#include <optional>
#include <string>
#include <vector>
#include "pulsatrix/text_tokenizer.hpp"
#include "pulsatrix/unicode_regex.hpp"
Include dependency graph for sentencepiece_bpe.hpp:

Go to the source code of this file.

Classes

class  pulsatrix::ReplaceNormalizer
 Replaces every match of a literal string or a regex (Hugging Face's Replace normalizer). More...
 
class  pulsatrix::PrependNormalizer
 Puts a string in front of non-empty text (Hugging Face's Prepend normalizer). More...
 
class  pulsatrix::MetaspacePreTokenizer
 Spaces become a replacement character (▁), one is put in front of each piece (or of the text's first piece only), and pieces are optionally split before each one (Hugging Face's Metaspace pre-tokenizer). More...
 
class  pulsatrix::ReplaceDecoder
 Replaces a literal string or regex in every token (Hugging Face's Replace decoder). More...
 
class  pulsatrix::ByteFallbackDecoder
 Turns runs of <0x41>-style byte tokens back into text; a run that isn't valid UTF-8 becomes one U+FFFD per byte (Hugging Face's ByteFallback decoder). More...
 
class  pulsatrix::StripDecoder
 Removes up to start leading and stop trailing copies of a character from every token (Hugging Face's Strip decoder). More...
 
class  pulsatrix::MetaspaceDecoder
 The replacement character back to spaces (Hugging Face's Metaspace decoder). Unless the scheme is Never, every replacement in the first token is dropped, as Hugging Face does, not only the one the pre-tokenizer put in front. More...
 
class  pulsatrix::SequenceDecoder
 Runs decoders in order, each on the previous one's tokens. More...
 

Namespaces

namespace  pulsatrix
 

Enumerations

enum class  pulsatrix::PrependScheme { pulsatrix::Always , pulsatrix::First , pulsatrix::Never }
 When Metaspace puts its replacement in front of a piece. More...
 

Detailed Description

The components SentencePiece-style BPE tokenizers are built from (TOK-3): spaces as ▁ (Replace, Prepend or Metaspace), byte fallback in BpeModel (BpeOptions::byte_fallback), and the decoder chain that undoes them, each following Hugging Face tokenizers. Gemma 3, Llama 2, Mistral and TinyLlama tokenize this way.