pulsatrix 1.1.0
Loading...
Searching...
No Matches
tokenizer_json.hpp
Go to the documentation of this file.
1
5#pragma once
6
7#include <string>
8#include <string_view>
9
11
12namespace pulsatrix {
13
30[[nodiscard]] TextTokenizer ParseTokenizerJson(std::string_view json);
31
33[[nodiscard]] TextTokenizer LoadTokenizerJson(const std::string& path);
34
35} // namespace pulsatrix
A tokenizer: added tokens are split out of the raw text first (longest match, earliest first); each r...
Definition text_tokenizer.hpp:166
Definition acquisition_functions.hpp:16
TextTokenizer LoadTokenizerJson(const std::string &path)
ParseTokenizerJson on a file.
TextTokenizer ParseTokenizerJson(std::string_view json)
Builds the tokenizer a tokenizer.json describes.
The tokenizer interface (TOK-1): a pipeline of normalizer, pre-tokenizer, model, post-processor and d...