pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::MetaspaceDecoder Class Reference

The replacement character back to spaces (Hugging Face's Metaspace decoder). Unless the scheme is Never, every replacement in the first token is dropped, as Hugging Face does, not only the one the pre-tokenizer put in front. More...

#include <sentencepiece_bpe.hpp>

Inheritance diagram for pulsatrix::MetaspaceDecoder:
Collaboration diagram for pulsatrix::MetaspaceDecoder:

Public Member Functions

 MetaspaceDecoder (std::string replacement, PrependScheme scheme)
 
std::vector< std::string > decode_chain (std::vector< std::string > tokens) const override
 The tokens after this decoder: by default decode()'s text as one token.
 
- Public Member Functions inherited from pulsatrix::Decoder
virtual ~Decoder ()=default
 
virtual std::string decode (const std::vector< std::string > &tokens) const
 The decoded text: decode_chain() joined.
 

Detailed Description

The replacement character back to spaces (Hugging Face's Metaspace decoder). Unless the scheme is Never, every replacement in the first token is dropped, as Hugging Face does, not only the one the pre-tokenizer put in front.

Constructor & Destructor Documentation

◆ MetaspaceDecoder()

pulsatrix::MetaspaceDecoder::MetaspaceDecoder ( std::string  replacement,
PrependScheme  scheme 
)

Member Function Documentation

◆ decode_chain()

std::vector< std::string > pulsatrix::MetaspaceDecoder::decode_chain ( std::vector< std::string >  tokens) const
overridevirtual

The tokens after this decoder: by default decode()'s text as one token.

Reimplemented from pulsatrix::Decoder.


The documentation for this class was generated from the following file: