pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::TemplatePostProcessor Class Reference

Special tokens around a single sequence (Hugging Face's TemplateProcessing, single). More...

#include <byte_level_bpe.hpp>

Inheritance diagram for pulsatrix::TemplatePostProcessor:
Collaboration diagram for pulsatrix::TemplatePostProcessor:

Classes

struct  Item
 One template item: the sequence itself (special empty), or special tokens. More...
 

Public Member Functions

 TemplatePostProcessor (std::vector< Item > single)
 
Encoding process (Encoding encoding, bool add_special_tokens) const override
 
- Public Member Functions inherited from pulsatrix::PostProcessor
virtual ~PostProcessor ()=default
 

Detailed Description

Special tokens around a single sequence (Hugging Face's TemplateProcessing, single).

Constructor & Destructor Documentation

◆ TemplatePostProcessor()

pulsatrix::TemplatePostProcessor::TemplatePostProcessor ( std::vector< Item >  single)
explicit

Member Function Documentation

◆ process()

Encoding pulsatrix::TemplatePostProcessor::process ( Encoding  encoding,
bool  add_special_tokens 
) const
overridevirtual

The documentation for this class was generated from the following file: