pulsatrix 1.1.0
Loading...
Searching...
No Matches
tokenizer_json.hpp File Reference

Loads a Hugging Face tokenizer.json into a TextTokenizer (TOK-2). More...

#include <string>
#include <string_view>
#include "pulsatrix/text_tokenizer.hpp"
Include dependency graph for tokenizer_json.hpp:

Go to the source code of this file.

Namespaces

namespace  pulsatrix
 

Functions

TextTokenizer pulsatrix::ParseTokenizerJson (std::string_view json)
 Builds the tokenizer a tokenizer.json describes.
 
TextTokenizer pulsatrix::LoadTokenizerJson (const std::string &path)
 ParseTokenizerJson on a file.
 

Detailed Description

Loads a Hugging Face tokenizer.json into a TextTokenizer (TOK-2).