A tokenizer: added tokens are split out of the raw text first (longest match, earliest first); each remaining segment is normalized, pre-tokenized and tokenized by the model; then the post-processor runs. Every component but the model is optional.
More...
#include <text_tokenizer.hpp>
A tokenizer: added tokens are split out of the raw text first (longest match, earliest first); each remaining segment is normalized, pre-tokenized and tokenized by the model; then the post-processor runs. Every component but the model is optional.
- Note
- Input must be valid UTF-8.
◆ TextTokenizer()
| pulsatrix::TextTokenizer::TextTokenizer |
( |
std::shared_ptr< const TokenModel > |
model | ) |
|
|
explicit |
◆ add_token()
- Exceptions
-
| std::invalid_argument | for empty content, or content or an id already added. |
◆ added_tokens()
| const std::vector< AddedToken > & pulsatrix::TextTokenizer::added_tokens |
( |
| ) |
const |
|
inline |
◆ decode()
| std::string pulsatrix::TextTokenizer::decode |
( |
const std::vector< int64_t > & |
ids, |
|
|
bool |
skip_special_tokens = false |
|
) |
| const |
The text for ids: their token strings through the decoder, or joined with spaces when there is none (as Hugging Face does).
- Exceptions
-
| std::out_of_range | for an id the tokenizer doesn't have. |
◆ encode()
| Encoding pulsatrix::TextTokenizer::encode |
( |
std::string_view |
text, |
|
|
bool |
add_special_tokens = true |
|
) |
| const |
- Exceptions
-
| std::invalid_argument | if text isn't valid UTF-8. |
◆ id_to_token()
| std::optional< std::string > pulsatrix::TextTokenizer::id_to_token |
( |
int64_t |
id | ) |
const |
◆ set_decoder()
◆ set_normalizer()
◆ set_post_processor()
◆ set_pre_tokenizer()
◆ token_to_id()
| std::optional< int64_t > pulsatrix::TextTokenizer::token_to_id |
( |
std::string_view |
token | ) |
const |
◆ vocab_size()
| int64_t pulsatrix::TextTokenizer::vocab_size |
( |
| ) |
const |
One more than the largest id, over the model and the added tokens.
The documentation for this class was generated from the following file: