|
pulsatrix 1.1.0
|
The components SentencePiece-style BPE tokenizers are built from (TOK-3): spaces as ▁ (Replace, Prepend or Metaspace), byte fallback in BpeModel (BpeOptions::byte_fallback), and the decoder chain that undoes them, each following Hugging Face tokenizers. Gemma 3, Llama 2, Mistral and TinyLlama tokenize this way.
More...
#include <memory>#include <optional>#include <string>#include <vector>#include "pulsatrix/text_tokenizer.hpp"#include "pulsatrix/unicode_regex.hpp"
Go to the source code of this file.
Classes | |
| class | pulsatrix::ReplaceNormalizer |
| Replaces every match of a literal string or a regex (Hugging Face's Replace normalizer). More... | |
| class | pulsatrix::PrependNormalizer |
| Puts a string in front of non-empty text (Hugging Face's Prepend normalizer). More... | |
| class | pulsatrix::MetaspacePreTokenizer |
Spaces become a replacement character (▁), one is put in front of each piece (or of the text's first piece only), and pieces are optionally split before each one (Hugging Face's Metaspace pre-tokenizer). More... | |
| class | pulsatrix::ReplaceDecoder |
| Replaces a literal string or regex in every token (Hugging Face's Replace decoder). More... | |
| class | pulsatrix::ByteFallbackDecoder |
Turns runs of <0x41>-style byte tokens back into text; a run that isn't valid UTF-8 becomes one U+FFFD per byte (Hugging Face's ByteFallback decoder). More... | |
| class | pulsatrix::StripDecoder |
Removes up to start leading and stop trailing copies of a character from every token (Hugging Face's Strip decoder). More... | |
| class | pulsatrix::MetaspaceDecoder |
| The replacement character back to spaces (Hugging Face's Metaspace decoder). Unless the scheme is Never, every replacement in the first token is dropped, as Hugging Face does, not only the one the pre-tokenizer put in front. More... | |
| class | pulsatrix::SequenceDecoder |
| Runs decoders in order, each on the previous one's tokens. More... | |
Namespaces | |
| namespace | pulsatrix |
Enumerations | |
| enum class | pulsatrix::PrependScheme { pulsatrix::Always , pulsatrix::First , pulsatrix::Never } |
| When Metaspace puts its replacement in front of a piece. More... | |
The components SentencePiece-style BPE tokenizers are built from (TOK-3): spaces as ▁ (Replace, Prepend or Metaspace), byte fallback in BpeModel (BpeOptions::byte_fallback), and the decoder chain that undoes them, each following Hugging Face tokenizers. Gemma 3, Llama 2, Mistral and TinyLlama tokenize this way.