pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::ByteLevelPostProcessor Class Reference

Byte-level post-processing (Hugging Face's ByteLevel post-processor): with trim_offsets, each token's offsets leave out the spaces at its edges. More...

#include <byte_level_bpe.hpp>

Inheritance diagram for pulsatrix::ByteLevelPostProcessor:
Collaboration diagram for pulsatrix::ByteLevelPostProcessor:

Public Member Functions

 ByteLevelPostProcessor (bool add_prefix_space, bool trim_offsets)
 
Encoding process (Encoding encoding, bool add_special_tokens) const override
 
- Public Member Functions inherited from pulsatrix::PostProcessor
virtual ~PostProcessor ()=default
 

Detailed Description

Byte-level post-processing (Hugging Face's ByteLevel post-processor): with trim_offsets, each token's offsets leave out the spaces at its edges.

Constructor & Destructor Documentation

◆ ByteLevelPostProcessor()

pulsatrix::ByteLevelPostProcessor::ByteLevelPostProcessor ( bool  add_prefix_space,
bool  trim_offsets 
)

Member Function Documentation

◆ process()

Encoding pulsatrix::ByteLevelPostProcessor::process ( Encoding  encoding,
bool  add_special_tokens 
) const
overridevirtual

The documentation for this class was generated from the following file: