30 ReplaceNormalizer(std::optional<UnicodeRegex> regex, std::string literal, std::string content);
31 std::optional<UnicodeRegex> regex_;
57 void pre_tokenize(std::vector<NormalizedString>& splits)
const override;
60 std::string replacement_;
70 [[nodiscard]] std::vector<std::string>
decode_chain(std::vector<std::string> tokens)
const override;
73 ReplaceDecoder(std::optional<UnicodeRegex> regex, std::string literal, std::string content);
74 std::optional<UnicodeRegex> regex_;
83 [[nodiscard]] std::vector<std::string>
decode_chain(std::vector<std::string> tokens)
const override;
91 [[nodiscard]] std::vector<std::string>
decode_chain(std::vector<std::string> tokens)
const override;
105 [[nodiscard]] std::vector<std::string>
decode_chain(std::vector<std::string> tokens)
const override;
108 std::string replacement_;
116 [[nodiscard]] std::vector<std::string>
decode_chain(std::vector<std::string> tokens)
const override;
119 std::vector<std::shared_ptr<const Decoder>> decoders_;
Turns runs of <0x41>-style byte tokens back into text; a run that isn't valid UTF-8 becomes one U+FFF...
Definition sentencepiece_bpe.hpp:81
std::vector< std::string > decode_chain(std::vector< std::string > tokens) const override
The tokens after this decoder: by default decode()'s text as one token.
Turns token strings back into text.
Definition text_tokenizer.hpp:143
Text being normalized and pre-tokenized, remembering where each byte came from in the original input,...
Definition text_tokenizer.hpp:55
Changes text before it is split (lowercasing, NFC, ...).
Definition text_tokenizer.hpp:96
Splits text into the pieces the model tokenizes separately, and may change them (byte- level mapping)...
Definition text_tokenizer.hpp:104
Puts a string in front of non-empty text (Hugging Face's Prepend normalizer).
Definition sentencepiece_bpe.hpp:37
PrependNormalizer(std::string prepend)
void normalize(NormalizedString &text) const override
Replaces a literal string or regex in every token (Hugging Face's Replace decoder).
Definition sentencepiece_bpe.hpp:66
ReplaceDecoder(std::string pattern, std::string content)
std::vector< std::string > decode_chain(std::vector< std::string > tokens) const override
The tokens after this decoder: by default decode()'s text as one token.
static ReplaceDecoder Regex(const std::string &pattern, std::string content)
Replaces every match of a literal string or a regex (Hugging Face's Replace normalizer).
Definition sentencepiece_bpe.hpp:21
ReplaceNormalizer(std::string pattern, std::string content)
A literal pattern.
void normalize(NormalizedString &text) const override
static ReplaceNormalizer Regex(const std::string &pattern, std::string content)
A regex pattern.
Runs decoders in order, each on the previous one's tokens.
Definition sentencepiece_bpe.hpp:113
std::vector< std::string > decode_chain(std::vector< std::string > tokens) const override
The tokens after this decoder: by default decode()'s text as one token.
SequenceDecoder(std::vector< std::shared_ptr< const Decoder > > decoders)
Removes up to start leading and stop trailing copies of a character from every token (Hugging Face's ...
Definition sentencepiece_bpe.hpp:88
StripDecoder(std::string content, size_t start, size_t stop)
std::vector< std::string > decode_chain(std::vector< std::string > tokens) const override
The tokens after this decoder: by default decode()'s text as one token.
Definition acquisition_functions.hpp:16
PrependScheme
When Metaspace puts its replacement in front of a piece.
Definition sentencepiece_bpe.hpp:47
The tokenizer interface (TOK-1): a pipeline of normalizer, pre-tokenizer, model, post-processor and d...
A small backtracking regex engine over Unicode code points, for the pre-tokenizer patterns in tokeniz...