pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::Encoding Struct Reference

The result of encoding one text. More...

#include <text_tokenizer.hpp>

Public Member Functions

size_t size () const
 
void push_back (int64_t id, std::string token, Offset offset, bool special)
 Appends one token.
 

Public Attributes

std::vector< int64_t > ids
 
std::vector< std::string > tokens
 
std::vector< Offset > offsets
 
std::vector< uint8_t > special_tokens_mask
 1 for special tokens: added tokens marked special, and tokens a post-processor inserted.
 

Detailed Description

The result of encoding one text.

Note
Offsets are UTF-8 byte offsets into the input, so text.substr(o.begin, o.end - o.begin) is the token's source. Hugging Face reports offsets in characters instead. A byte-level token can cover part of a multi-byte character, and then its offset covers just those bytes. A character a normalizer composed (NFC) maps to all of its source characters, where Hugging Face maps it to the first only. Tokens inserted by a post-processor (BOS, EOS) have an empty offset.

Member Function Documentation

◆ push_back()

void pulsatrix::Encoding::push_back ( int64_t  id,
std::string  token,
Offset  offset,
bool  special 
)

Appends one token.

◆ size()

size_t pulsatrix::Encoding::size ( ) const
inline

Member Data Documentation

◆ ids

std::vector<int64_t> pulsatrix::Encoding::ids

◆ offsets

std::vector<Offset> pulsatrix::Encoding::offsets

◆ special_tokens_mask

std::vector<uint8_t> pulsatrix::Encoding::special_tokens_mask

1 for special tokens: added tokens marked special, and tokens a post-processor inserted.

◆ tokens

std::vector<std::string> pulsatrix::Encoding::tokens

The documentation for this struct was generated from the following file: