pulsatrix 1.1.0
Loading...
Searching...
No Matches
byte_level_bpe.hpp File Reference

The components byte-level BPE tokenizers are built from (TOK-2): NFC, regex and digit splitting, byte-level mapping, BPE, template post-processing and byte-level decoding, each following Hugging Face tokenizers. More...

#include <cstdint>
#include <memory>
#include <optional>
#include <string>
#include <unordered_map>
#include <utility>
#include <vector>
#include "pulsatrix/text_tokenizer.hpp"
#include "pulsatrix/unicode_regex.hpp"
Include dependency graph for byte_level_bpe.hpp:

Go to the source code of this file.

Classes

class  pulsatrix::NfcNormalizer
 Unicode Normalization Form C (canonical composition), keeping offsets. More...
 
class  pulsatrix::SequenceNormalizer
 Runs normalizers in order. More...
 
class  pulsatrix::SplitPreTokenizer
 Splits on a regex or a literal string (Hugging Face's Split pre-tokenizer). More...
 
class  pulsatrix::DigitsPreTokenizer
 Splits off numeric characters (Unicode N), one by one or in runs (Hugging Face's Digits). More...
 
class  pulsatrix::ByteLevelPreTokenizer
 Byte-level pre-tokenization (Hugging Face's ByteLevel): optionally prepends a space, optionally splits with the GPT-2 regex, then maps every byte to a printable character with GPT-2's table, so BPE runs over characters that stand for bytes. More...
 
class  pulsatrix::SequencePreTokenizer
 Runs pre-tokenizers in order. More...
 
struct  pulsatrix::BpeOptions
 BPE options beyond the vocabulary and merges, as tokenizer.json names them. More...
 
class  pulsatrix::BpeModel
 Byte-pair encoding: start from characters and apply merges, lowest rank first. More...
 
class  pulsatrix::ByteLevelPostProcessor
 Byte-level post-processing (Hugging Face's ByteLevel post-processor): with trim_offsets, each token's offsets leave out the spaces at its edges. More...
 
class  pulsatrix::TemplatePostProcessor
 Special tokens around a single sequence (Hugging Face's TemplateProcessing, single). More...
 
struct  pulsatrix::TemplatePostProcessor::Item
 One template item: the sequence itself (special empty), or special tokens. More...
 
class  pulsatrix::SequencePostProcessor
 Runs post-processors in order. More...
 
class  pulsatrix::ByteLevelDecoder
 Maps byte-level characters back to bytes; invalid UTF-8 becomes U+FFFD, as in Hugging Face. More...
 

Namespaces

namespace  pulsatrix
 

Enumerations

enum class  pulsatrix::SplitBehavior {
  pulsatrix::Removed , pulsatrix::Isolated , pulsatrix::MergedWithPrevious , pulsatrix::MergedWithNext ,
  pulsatrix::Contiguous
}
 What a split does with the pieces that match (Hugging Face's SplitDelimiterBehavior). More...
 

Functions

const std::vector< std::string > & pulsatrix::ByteLevelAlphabet ()
 The GPT-2 byte-to-character table byte-level BPE uses: printable bytes stand for themselves, the rest for U+0100 onward.
 

Detailed Description

The components byte-level BPE tokenizers are built from (TOK-2): NFC, regex and digit splitting, byte-level mapping, BPE, template post-processing and byte-level decoding, each following Hugging Face tokenizers.