pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::TextTokenizer Class Reference

A tokenizer: added tokens are split out of the raw text first (longest match, earliest first); each remaining segment is normalized, pre-tokenized and tokenized by the model; then the post-processor runs. Every component but the model is optional. More...

#include <text_tokenizer.hpp>

Public Member Functions

 TextTokenizer (std::shared_ptr< const TokenModel > model)
 
TextTokenizer & set_normalizer (std::shared_ptr< const Normalizer > normalizer)
 
TextTokenizer & set_pre_tokenizer (std::shared_ptr< const PreTokenizer > pre_tokenizer)
 
TextTokenizer & set_post_processor (std::shared_ptr< const PostProcessor > post_processor)
 
TextTokenizer & set_decoder (std::shared_ptr< const Decoder > decoder)
 
TextTokenizer & add_token (AddedToken token)
 
Encoding encode (std::string_view text, bool add_special_tokens=true) const
 
std::string decode (const std::vector< int64_t > &ids, bool skip_special_tokens=false) const
 The text for ids: their token strings through the decoder, or joined with spaces when there is none (as Hugging Face does).
 
int64_t vocab_size () const
 One more than the largest id, over the model and the added tokens.
 
std::optional< int64_t > token_to_id (std::string_view token) const
 
std::optional< std::string > id_to_token (int64_t id) const
 
const std::vector< AddedToken > & added_tokens () const
 

Detailed Description

A tokenizer: added tokens are split out of the raw text first (longest match, earliest first); each remaining segment is normalized, pre-tokenized and tokenized by the model; then the post-processor runs. Every component but the model is optional.

Note
Input must be valid UTF-8.

Constructor & Destructor Documentation

◆ TextTokenizer()

pulsatrix::TextTokenizer::TextTokenizer ( std::shared_ptr< const TokenModel >  model)
explicit

Member Function Documentation

◆ add_token()

TextTokenizer & pulsatrix::TextTokenizer::add_token ( AddedToken  token)
Exceptions
std::invalid_argumentfor empty content, or content or an id already added.

◆ added_tokens()

const std::vector< AddedToken > & pulsatrix::TextTokenizer::added_tokens ( ) const
inline

◆ decode()

std::string pulsatrix::TextTokenizer::decode ( const std::vector< int64_t > &  ids,
bool  skip_special_tokens = false 
) const

The text for ids: their token strings through the decoder, or joined with spaces when there is none (as Hugging Face does).

Exceptions
std::out_of_rangefor an id the tokenizer doesn't have.

◆ encode()

Encoding pulsatrix::TextTokenizer::encode ( std::string_view  text,
bool  add_special_tokens = true 
) const
Exceptions
std::invalid_argumentif text isn't valid UTF-8.

◆ id_to_token()

std::optional< std::string > pulsatrix::TextTokenizer::id_to_token ( int64_t  id) const

◆ set_decoder()

TextTokenizer & pulsatrix::TextTokenizer::set_decoder ( std::shared_ptr< const Decoder >  decoder)

◆ set_normalizer()

TextTokenizer & pulsatrix::TextTokenizer::set_normalizer ( std::shared_ptr< const Normalizer >  normalizer)

◆ set_post_processor()

TextTokenizer & pulsatrix::TextTokenizer::set_post_processor ( std::shared_ptr< const PostProcessor >  post_processor)

◆ set_pre_tokenizer()

TextTokenizer & pulsatrix::TextTokenizer::set_pre_tokenizer ( std::shared_ptr< const PreTokenizer >  pre_tokenizer)

◆ token_to_id()

std::optional< int64_t > pulsatrix::TextTokenizer::token_to_id ( std::string_view  token) const

◆ vocab_size()

int64_t pulsatrix::TextTokenizer::vocab_size ( ) const

One more than the largest id, over the model and the added tokens.


The documentation for this class was generated from the following file: