|
pulsatrix 1.1.0
|
One token from a TokenModel: its id and string, and the bytes of the piece it covers. More...
#include <text_tokenizer.hpp>
Public Attributes | |
| int64_t | id = 0 |
| std::string | value |
| size_t | begin = 0 |
| size_t | end = 0 |
One token from a TokenModel: its id and string, and the bytes of the piece it covers.
| size_t pulsatrix::ModelToken::begin = 0 |
| size_t pulsatrix::ModelToken::end = 0 |
| int64_t pulsatrix::ModelToken::id = 0 |
| std::string pulsatrix::ModelToken::value |