|
pulsatrix 1.1.0
|
Every byte is a token whose id is the byte's value (0..255); the token string is the byte itself. More...
#include <tokenizer_components.hpp>


Public Member Functions | |
| std::vector< ModelToken > | tokenize (std::string_view piece) const override |
| std::optional< int64_t > | token_to_id (std::string_view token) const override |
| std::optional< std::string > | id_to_token (int64_t id) const override |
| int64_t | vocab_size () const override |
| One more than the largest id the model can produce. | |
Public Member Functions inherited from pulsatrix::TokenModel | |
| virtual | ~TokenModel ()=default |
Every byte is a token whose id is the byte's value (0..255); the token string is the byte itself.
|
overridevirtual |
Implements pulsatrix::TokenModel.
|
overridevirtual |
Implements pulsatrix::TokenModel.
|
overridevirtual |
Implements pulsatrix::TokenModel.
|
inlineoverridevirtual |
One more than the largest id the model can produce.
Implements pulsatrix::TokenModel.