|
pulsatrix 1.1.0
|
Turns one pre-tokenized piece into tokens (word-level lookup, BPE, ...). More...
#include <text_tokenizer.hpp>

Public Member Functions | |
| virtual | ~TokenModel ()=default |
| virtual std::vector< ModelToken > | tokenize (std::string_view piece) const =0 |
| virtual std::optional< int64_t > | token_to_id (std::string_view token) const =0 |
| virtual std::optional< std::string > | id_to_token (int64_t id) const =0 |
| virtual int64_t | vocab_size () const =0 |
| One more than the largest id the model can produce. | |
Turns one pre-tokenized piece into tokens (word-level lookup, BPE, ...).
|
virtualdefault |
|
pure virtual |
Implemented in pulsatrix::BpeModel, pulsatrix::WordLevelModel, pulsatrix::ByteModel, and pulsatrix::CharModel.
|
pure virtual |
Implemented in pulsatrix::BpeModel, pulsatrix::WordLevelModel, pulsatrix::ByteModel, and pulsatrix::CharModel.
|
pure virtual |
Implemented in pulsatrix::BpeModel, pulsatrix::WordLevelModel, pulsatrix::ByteModel, and pulsatrix::CharModel.
|
pure virtual |
One more than the largest id the model can produce.
Implemented in pulsatrix::BpeModel, pulsatrix::WordLevelModel, pulsatrix::ByteModel, and pulsatrix::CharModel.