|
pulsatrix 1.1.0
|
Special tokens around a single sequence (Hugging Face's TemplateProcessing, single).
More...
#include <byte_level_bpe.hpp>


Classes | |
| struct | Item |
One template item: the sequence itself (special empty), or special tokens. More... | |
Public Member Functions | |
| TemplatePostProcessor (std::vector< Item > single) | |
| Encoding | process (Encoding encoding, bool add_special_tokens) const override |
Public Member Functions inherited from pulsatrix::PostProcessor | |
| virtual | ~PostProcessor ()=default |
Special tokens around a single sequence (Hugging Face's TemplateProcessing, single).
|
explicit |
|
overridevirtual |
Implements pulsatrix::PostProcessor.