|
pulsatrix 1.1.0
|
Loads a Hugging Face tokenizer.json into a TextTokenizer (TOK-2).
More...

Go to the source code of this file.
Namespaces | |
| namespace | pulsatrix |
Functions | |
| TextTokenizer | pulsatrix::ParseTokenizerJson (std::string_view json) |
Builds the tokenizer a tokenizer.json describes. | |
| TextTokenizer | pulsatrix::LoadTokenizerJson (const std::string &path) |
| ParseTokenizerJson on a file. | |
Loads a Hugging Face tokenizer.json into a TextTokenizer (TOK-2).