|
pulsatrix 1.1.0
|
Byte-level post-processing (Hugging Face's ByteLevel post-processor): with trim_offsets, each token's offsets leave out the spaces at its edges. More...
#include <byte_level_bpe.hpp>


Public Member Functions | |
| ByteLevelPostProcessor (bool add_prefix_space, bool trim_offsets) | |
| Encoding | process (Encoding encoding, bool add_special_tokens) const override |
Public Member Functions inherited from pulsatrix::PostProcessor | |
| virtual | ~PostProcessor ()=default |
Byte-level post-processing (Hugging Face's ByteLevel post-processor): with trim_offsets, each token's offsets leave out the spaces at its edges.
| pulsatrix::ByteLevelPostProcessor::ByteLevelPostProcessor | ( | bool | add_prefix_space, |
| bool | trim_offsets | ||
| ) |
|
overridevirtual |
Implements pulsatrix::PostProcessor.