pulsatrix 1.1.0
Loading...
Searching...
No Matches
text_tokenizer.hpp File Reference

The tokenizer interface (TOK-1): a pipeline of normalizer, pre-tokenizer, model, post-processor and decoder, as in Hugging Face tokenizers, with added tokens split out first. encode() returns ids, token strings, offsets into the input and a special- token mask; decode() returns text. More...

#include <cstddef>
#include <cstdint>
#include <memory>
#include <optional>
#include <string>
#include <string_view>
#include <unordered_map>
#include <vector>
Include dependency graph for text_tokenizer.hpp:
This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes

struct  pulsatrix::Offset
 A byte range [begin, end) of the text passed to TextTokenizer::encode(). More...
 
struct  pulsatrix::Encoding
 The result of encoding one text. More...
 
class  pulsatrix::NormalizedString
 Text being normalized and pre-tokenized, remembering where each byte came from in the original input, so offsets survive changes such as NFC composition or byte-to-character mapping. More...
 
struct  pulsatrix::NormalizedString::Piece
 One piece of a rebuilt text: its new bytes, and the range of the current text they replace (an empty range inserts). More...
 
class  pulsatrix::Normalizer
 Changes text before it is split (lowercasing, NFC, ...). More...
 
class  pulsatrix::PreTokenizer
 Splits text into the pieces the model tokenizes separately, and may change them (byte- level mapping). More...
 
struct  pulsatrix::ModelToken
 One token from a TokenModel: its id and string, and the bytes of the piece it covers. More...
 
class  pulsatrix::TokenModel
 Turns one pre-tokenized piece into tokens (word-level lookup, BPE, ...). More...
 
class  pulsatrix::PostProcessor
 Adds tokens around an encoding (BOS and EOS, sentence pairs, ...). More...
 
class  pulsatrix::Decoder
 Turns token strings back into text. More...
 
struct  pulsatrix::AddedToken
 A token matched in the raw input before anything else runs, such as <|im_start|>. More...
 
class  pulsatrix::TextTokenizer
 A tokenizer: added tokens are split out of the raw text first (longest match, earliest first); each remaining segment is normalized, pre-tokenized and tokenized by the model; then the post-processor runs. Every component but the model is optional. More...
 

Namespaces

namespace  pulsatrix
 

Detailed Description

The tokenizer interface (TOK-1): a pipeline of normalizer, pre-tokenizer, model, post-processor and decoder, as in Hugging Face tokenizers, with added tokens split out first. encode() returns ids, token strings, offsets into the input and a special- token mask; decode() returns text.