57 size_t max_examples = 5);
A tokenizer: added tokens are split out of the raw text first (longest match, earliest first); each r...
Definition text_tokenizer.hpp:166
Definition acquisition_functions.hpp:16
TokenizerParityReport CompareToTokenizerReference(const TextTokenizer &tokenizer, const std::string &path, size_t max_examples=5)
Encodes and decodes every line of a reference file (JSON lines with text, ids, plain_ids,...
Offset ByteToCharOffset(std::string_view text, Offset bytes)
A byte offset range of text as a character (code point) range, widened to whole characters,...
A byte range [begin, end) of the text passed to TextTokenizer::encode().
Definition text_tokenizer.hpp:22
The first difference on one line of the corpus.
Definition tokenizer_parity.hpp:17
size_t line
Definition tokenizer_parity.hpp:18
std::string field
"ids", "plain_ids", "offsets", "decoded" or "decoded_skip".
Definition tokenizer_parity.hpp:20
std::string detail
Definition tokenizer_parity.hpp:22
std::string text
Definition tokenizer_parity.hpp:21
How a tokenizer compares with the reference, field by field.
Definition tokenizer_parity.hpp:26
bool passed() const
Definition tokenizer_parity.hpp:43
std::vector< TokenizerMismatch > examples
The first few mismatches, for diagnosis.
Definition tokenizer_parity.hpp:42
size_t lines
Definition tokenizer_parity.hpp:27
size_t decode_mismatches
lines whose decoded text differs
Definition tokenizer_parity.hpp:40
size_t inexact_line_offset_differences
Lines that don't round-trip exactly whose offsets differ: counted, not failed. Such a line was change...
Definition tokenizer_parity.hpp:39
size_t offset_mismatches
Lines that round-trip exactly whose ids match but offsets don't.
Definition tokenizer_parity.hpp:30
size_t id_mismatches
lines whose ids (with or without special tokens) differ
Definition tokenizer_parity.hpp:28
The tokenizer interface (TOK-1): a pipeline of normalizer, pre-tokenizer, model, post-processor and d...