pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::ByteLevelDecoder Class Reference

Maps byte-level characters back to bytes; invalid UTF-8 becomes U+FFFD, as in Hugging Face. More...

#include <byte_level_bpe.hpp>

Inheritance diagram for pulsatrix::ByteLevelDecoder:
Collaboration diagram for pulsatrix::ByteLevelDecoder:

Public Member Functions

std::string decode (const std::vector< std::string > &tokens) const override
 The decoded text: decode_chain() joined.
 
- Public Member Functions inherited from pulsatrix::Decoder
virtual ~Decoder ()=default
 
virtual std::vector< std::string > decode_chain (std::vector< std::string > tokens) const
 The tokens after this decoder: by default decode()'s text as one token.
 

Detailed Description

Maps byte-level characters back to bytes; invalid UTF-8 becomes U+FFFD, as in Hugging Face.

Member Function Documentation

◆ decode()

std::string pulsatrix::ByteLevelDecoder::decode ( const std::vector< std::string > &  tokens) const
overridevirtual

The decoded text: decode_chain() joined.

Reimplemented from pulsatrix::Decoder.


The documentation for this class was generated from the following file: