pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::BpeOptions Struct Reference

BPE options beyond the vocabulary and merges, as tokenizer.json names them. More...

#include <byte_level_bpe.hpp>

Public Attributes

bool ignore_merges = false
 A piece that is a whole vocabulary entry is one token, without merging (Llama 3).
 
std::string continuing_subword_prefix
 Added before every character but a piece's first.
 
std::string end_of_word_suffix
 Added after a piece's last character.
 
std::optional< std::string > unk_token
 For characters the vocabulary lacks; without one they are dropped, as in Hugging Face.
 
bool byte_fallback = false
 A character the vocabulary lacks becomes its UTF-8 bytes as <0x41>-style tokens (SentencePiece's byte fallback), before the unknown token is considered.
 
bool fuse_unk = false
 Consecutive unknown characters become one unknown token.
 

Detailed Description

BPE options beyond the vocabulary and merges, as tokenizer.json names them.

Member Data Documentation

◆ byte_fallback

bool pulsatrix::BpeOptions::byte_fallback = false

A character the vocabulary lacks becomes its UTF-8 bytes as <0x41>-style tokens (SentencePiece's byte fallback), before the unknown token is considered.

◆ continuing_subword_prefix

std::string pulsatrix::BpeOptions::continuing_subword_prefix

Added before every character but a piece's first.

◆ end_of_word_suffix

std::string pulsatrix::BpeOptions::end_of_word_suffix

Added after a piece's last character.

◆ fuse_unk

bool pulsatrix::BpeOptions::fuse_unk = false

Consecutive unknown characters become one unknown token.

◆ ignore_merges

bool pulsatrix::BpeOptions::ignore_merges = false

A piece that is a whole vocabulary entry is one token, without merging (Llama 3).

◆ unk_token

std::optional<std::string> pulsatrix::BpeOptions::unk_token

For characters the vocabulary lacks; without one they are dropped, as in Hugging Face.


The documentation for this struct was generated from the following file: