pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::Decoder Class Reference

Turns token strings back into text. More...

#include <text_tokenizer.hpp>

Inheritance diagram for pulsatrix::Decoder:

Public Member Functions

virtual ~Decoder ()=default
 
virtual std::string decode (const std::vector< std::string > &tokens) const
 The decoded text: decode_chain() joined.
 
virtual std::vector< std::string > decode_chain (std::vector< std::string > tokens) const
 The tokens after this decoder: by default decode()'s text as one token.
 

Detailed Description

Turns token strings back into text.

Turns token strings back into text. As in Hugging Face, a decoder maps a token list to a token list (decode_chain), so decoders compose (SequenceDecoder), and decode() joins the result. Override at least one of the two.

Constructor & Destructor Documentation

◆ ~Decoder()

virtual pulsatrix::Decoder::~Decoder ( )
virtualdefault

Member Function Documentation

◆ decode()

virtual std::string pulsatrix::Decoder::decode ( const std::vector< std::string > &  tokens) const
virtual

The decoded text: decode_chain() joined.

Reimplemented in pulsatrix::ByteLevelDecoder, and pulsatrix::FuseDecoder.

◆ decode_chain()

virtual std::vector< std::string > pulsatrix::Decoder::decode_chain ( std::vector< std::string >  tokens) const
virtual

The tokens after this decoder: by default decode()'s text as one token.

Reimplemented in pulsatrix::ReplaceDecoder, pulsatrix::ByteFallbackDecoder, pulsatrix::StripDecoder, pulsatrix::MetaspaceDecoder, and pulsatrix::SequenceDecoder.


The documentation for this class was generated from the following file: