|
pulsatrix 1.1.0
|
A token matched in the raw input before anything else runs, such as <|im_start|>.
More...
#include <text_tokenizer.hpp>
Public Attributes | |
| std::string | content |
| int64_t | id = 0 |
| bool | special = true |
| Marked in the special-tokens mask and dropped by decode(skip_special_tokens). | |
A token matched in the raw input before anything else runs, such as <|im_start|>.
| std::string pulsatrix::AddedToken::content |
| int64_t pulsatrix::AddedToken::id = 0 |
| bool pulsatrix::AddedToken::special = true |
Marked in the special-tokens mask and dropped by decode(skip_special_tokens).