pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::ModelToken Struct Reference

One token from a TokenModel: its id and string, and the bytes of the piece it covers. More...

#include <text_tokenizer.hpp>

Public Attributes

int64_t id = 0
 
std::string value
 
size_t begin = 0
 
size_t end = 0
 

Detailed Description

One token from a TokenModel: its id and string, and the bytes of the piece it covers.

Member Data Documentation

◆ begin

size_t pulsatrix::ModelToken::begin = 0

◆ end

size_t pulsatrix::ModelToken::end = 0

◆ id

int64_t pulsatrix::ModelToken::id = 0

◆ value

std::string pulsatrix::ModelToken::value

The documentation for this struct was generated from the following file: