|
pulsatrix 1.1.0
|
Adds tokens around an encoding (BOS and EOS, sentence pairs, ...). More...
#include <text_tokenizer.hpp>

Public Member Functions | |
| virtual | ~PostProcessor ()=default |
| virtual Encoding | process (Encoding encoding, bool add_special_tokens) const =0 |
Adds tokens around an encoding (BOS and EOS, sentence pairs, ...).
|
virtualdefault |