pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::TokenModel Class Referenceabstract

Turns one pre-tokenized piece into tokens (word-level lookup, BPE, ...). More...

#include <text_tokenizer.hpp>

Inheritance diagram for pulsatrix::TokenModel:

Public Member Functions

virtual ~TokenModel ()=default
 
virtual std::vector< ModelToken > tokenize (std::string_view piece) const =0
 
virtual std::optional< int64_t > token_to_id (std::string_view token) const =0
 
virtual std::optional< std::string > id_to_token (int64_t id) const =0
 
virtual int64_t vocab_size () const =0
 One more than the largest id the model can produce.
 

Detailed Description

Turns one pre-tokenized piece into tokens (word-level lookup, BPE, ...).

Constructor & Destructor Documentation

◆ ~TokenModel()

virtual pulsatrix::TokenModel::~TokenModel ( )
virtualdefault

Member Function Documentation

◆ id_to_token()

virtual std::optional< std::string > pulsatrix::TokenModel::id_to_token ( int64_t  id) const
pure virtual

◆ token_to_id()

virtual std::optional< int64_t > pulsatrix::TokenModel::token_to_id ( std::string_view  token) const
pure virtual

◆ tokenize()

virtual std::vector< ModelToken > pulsatrix::TokenModel::tokenize ( std::string_view  piece) const
pure virtual

◆ vocab_size()

virtual int64_t pulsatrix::TokenModel::vocab_size ( ) const
pure virtual

One more than the largest id the model can produce.

Implemented in pulsatrix::BpeModel, pulsatrix::WordLevelModel, pulsatrix::ByteModel, and pulsatrix::CharModel.


The documentation for this class was generated from the following file: