|
pulsatrix 1.1.0
|
BPE options beyond the vocabulary and merges, as tokenizer.json names them.
More...
#include <byte_level_bpe.hpp>
Public Attributes | |
| bool | ignore_merges = false |
| A piece that is a whole vocabulary entry is one token, without merging (Llama 3). | |
| std::string | continuing_subword_prefix |
| Added before every character but a piece's first. | |
| std::string | end_of_word_suffix |
| Added after a piece's last character. | |
| std::optional< std::string > | unk_token |
| For characters the vocabulary lacks; without one they are dropped, as in Hugging Face. | |
| bool | byte_fallback = false |
A character the vocabulary lacks becomes its UTF-8 bytes as <0x41>-style tokens (SentencePiece's byte fallback), before the unknown token is considered. | |
| bool | fuse_unk = false |
| Consecutive unknown characters become one unknown token. | |
BPE options beyond the vocabulary and merges, as tokenizer.json names them.
| bool pulsatrix::BpeOptions::byte_fallback = false |
A character the vocabulary lacks becomes its UTF-8 bytes as <0x41>-style tokens (SentencePiece's byte fallback), before the unknown token is considered.
| std::string pulsatrix::BpeOptions::continuing_subword_prefix |
Added before every character but a piece's first.
| std::string pulsatrix::BpeOptions::end_of_word_suffix |
Added after a piece's last character.
| bool pulsatrix::BpeOptions::fuse_unk = false |
Consecutive unknown characters become one unknown token.
| bool pulsatrix::BpeOptions::ignore_merges = false |
A piece that is a whole vocabulary entry is one token, without merging (Llama 3).
| std::optional<std::string> pulsatrix::BpeOptions::unk_token |
For characters the vocabulary lacks; without one they are dropped, as in Hugging Face.