pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::AddedToken Struct Reference

A token matched in the raw input before anything else runs, such as <|im_start|>. More...

#include <text_tokenizer.hpp>

Public Attributes

std::string content
 
int64_t id = 0
 
bool special = true
 Marked in the special-tokens mask and dropped by decode(skip_special_tokens).
 

Detailed Description

A token matched in the raw input before anything else runs, such as <|im_start|>.

Member Data Documentation

◆ content

std::string pulsatrix::AddedToken::content

◆ id

int64_t pulsatrix::AddedToken::id = 0

◆ special

bool pulsatrix::AddedToken::special = true

Marked in the special-tokens mask and dropped by decode(skip_special_tokens).


The documentation for this struct was generated from the following file: