pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::BpeModel Class Reference

Byte-pair encoding: start from characters and apply merges, lowest rank first. More...

#include <byte_level_bpe.hpp>

Inheritance diagram for pulsatrix::BpeModel:
Collaboration diagram for pulsatrix::BpeModel:

Public Member Functions

 BpeModel (std::unordered_map< std::string, int64_t > vocab, const std::vector< std::pair< std::string, std::string > > &merges, BpeOptions options={})
 
std::vector< ModelToken > tokenize (std::string_view piece) const override
 
std::optional< int64_t > token_to_id (std::string_view token) const override
 
std::optional< std::string > id_to_token (int64_t id) const override
 
int64_t vocab_size () const override
 One more than the largest id the model can produce.
 
- Public Member Functions inherited from pulsatrix::TokenModel
virtual ~TokenModel ()=default
 

Detailed Description

Byte-pair encoding: start from characters and apply merges, lowest rank first.

Constructor & Destructor Documentation

◆ BpeModel()

pulsatrix::BpeModel::BpeModel ( std::unordered_map< std::string, int64_t >  vocab,
const std::vector< std::pair< std::string, std::string > > &  merges,
BpeOptions  options = {} 
)
Parameters
vocabToken strings and their ids.
mergesMerge rules, highest priority first.
Exceptions
std::invalid_argumentif a merge's parts or result aren't in the vocabulary, or the unknown token isn't.

Member Function Documentation

◆ id_to_token()

std::optional< std::string > pulsatrix::BpeModel::id_to_token ( int64_t  id) const
overridevirtual

Implements pulsatrix::TokenModel.

◆ token_to_id()

std::optional< int64_t > pulsatrix::BpeModel::token_to_id ( std::string_view  token) const
overridevirtual

Implements pulsatrix::TokenModel.

◆ tokenize()

std::vector< ModelToken > pulsatrix::BpeModel::tokenize ( std::string_view  piece) const
overridevirtual

Implements pulsatrix::TokenModel.

◆ vocab_size()

int64_t pulsatrix::BpeModel::vocab_size ( ) const
inlineoverridevirtual

One more than the largest id the model can produce.

Implements pulsatrix::TokenModel.


The documentation for this class was generated from the following file: