The result of encoding one text.
More...
#include <text_tokenizer.hpp>
|
| size_t | size () const |
| |
| void | push_back (int64_t id, std::string token, Offset offset, bool special) |
| | Appends one token.
|
| |
|
| std::vector< int64_t > | ids |
| |
| std::vector< std::string > | tokens |
| |
| std::vector< Offset > | offsets |
| |
| std::vector< uint8_t > | special_tokens_mask |
| | 1 for special tokens: added tokens marked special, and tokens a post-processor inserted.
|
| |
The result of encoding one text.
- Note
- Offsets are UTF-8 byte offsets into the input, so
text.substr(o.begin, o.end - o.begin) is the token's source. Hugging Face reports offsets in characters instead. A byte-level token can cover part of a multi-byte character, and then its offset covers just those bytes. A character a normalizer composed (NFC) maps to all of its source characters, where Hugging Face maps it to the first only. Tokens inserted by a post-processor (BOS, EOS) have an empty offset.
◆ push_back()
| void pulsatrix::Encoding::push_back |
( |
int64_t |
id, |
|
|
std::string |
token, |
|
|
Offset |
offset, |
|
|
bool |
special |
|
) |
| |
◆ size()
| size_t pulsatrix::Encoding::size |
( |
| ) |
const |
|
inline |
◆ ids
| std::vector<int64_t> pulsatrix::Encoding::ids |
◆ offsets
| std::vector<Offset> pulsatrix::Encoding::offsets |
◆ special_tokens_mask
| std::vector<uint8_t> pulsatrix::Encoding::special_tokens_mask |
1 for special tokens: added tokens marked special, and tokens a post-processor inserted.
◆ tokens
| std::vector<std::string> pulsatrix::Encoding::tokens |
The documentation for this struct was generated from the following file: