pulsatrix
1.1.0
Loading...
Searching...
No Matches
tokenizer_json.hpp
Go to the documentation of this file.
1
5
#pragma once
6
7
#include <string>
8
#include <string_view>
9
10
#include "
pulsatrix/text_tokenizer.hpp
"
11
12
namespace
pulsatrix
{
13
30
[[nodiscard]]
TextTokenizer
ParseTokenizerJson
(std::string_view json);
31
33
[[nodiscard]]
TextTokenizer
LoadTokenizerJson
(
const
std::string& path);
34
35
}
// namespace pulsatrix
pulsatrix::TextTokenizer
A tokenizer: added tokens are split out of the raw text first (longest match, earliest first); each r...
Definition
text_tokenizer.hpp:166
pulsatrix
Definition
acquisition_functions.hpp:16
pulsatrix::LoadTokenizerJson
TextTokenizer LoadTokenizerJson(const std::string &path)
ParseTokenizerJson on a file.
pulsatrix::ParseTokenizerJson
TextTokenizer ParseTokenizerJson(std::string_view json)
Builds the tokenizer a tokenizer.json describes.
text_tokenizer.hpp
The tokenizer interface (TOK-1): a pipeline of normalizer, pre-tokenizer, model, post-processor and d...
include
pulsatrix
tokenizer_json.hpp
Generated by
1.9.8