pulsatrix 1.1.0
Loading...
Searching...
No Matches
tokenizer_parity.hpp
Go to the documentation of this file.
1
6#pragma once
7
8#include <cstddef>
9#include <string>
10#include <vector>
11
13
14namespace pulsatrix {
15
18 size_t line = 0;
20 std::string field;
21 std::string text;
22 std::string detail;
23};
24
27 size_t lines = 0;
28 size_t id_mismatches = 0;
40 size_t decode_mismatches = 0;
42 std::vector<TokenizerMismatch> examples;
43 [[nodiscard]] bool passed() const { return lines > 0 && id_mismatches == 0 && offset_mismatches == 0 && decode_mismatches == 0; }
44};
45
56[[nodiscard]] TokenizerParityReport CompareToTokenizerReference(const TextTokenizer& tokenizer, const std::string& path,
57 size_t max_examples = 5);
58
61[[nodiscard]] Offset ByteToCharOffset(std::string_view text, Offset bytes);
62
63} // namespace pulsatrix
A tokenizer: added tokens are split out of the raw text first (longest match, earliest first); each r...
Definition text_tokenizer.hpp:166
Definition acquisition_functions.hpp:16
TokenizerParityReport CompareToTokenizerReference(const TextTokenizer &tokenizer, const std::string &path, size_t max_examples=5)
Encodes and decodes every line of a reference file (JSON lines with text, ids, plain_ids,...
Offset ByteToCharOffset(std::string_view text, Offset bytes)
A byte offset range of text as a character (code point) range, widened to whole characters,...
A byte range [begin, end) of the text passed to TextTokenizer::encode().
Definition text_tokenizer.hpp:22
The first difference on one line of the corpus.
Definition tokenizer_parity.hpp:17
size_t line
Definition tokenizer_parity.hpp:18
std::string field
"ids", "plain_ids", "offsets", "decoded" or "decoded_skip".
Definition tokenizer_parity.hpp:20
std::string detail
Definition tokenizer_parity.hpp:22
std::string text
Definition tokenizer_parity.hpp:21
How a tokenizer compares with the reference, field by field.
Definition tokenizer_parity.hpp:26
bool passed() const
Definition tokenizer_parity.hpp:43
std::vector< TokenizerMismatch > examples
The first few mismatches, for diagnosis.
Definition tokenizer_parity.hpp:42
size_t lines
Definition tokenizer_parity.hpp:27
size_t decode_mismatches
lines whose decoded text differs
Definition tokenizer_parity.hpp:40
size_t inexact_line_offset_differences
Lines that don't round-trip exactly whose offsets differ: counted, not failed. Such a line was change...
Definition tokenizer_parity.hpp:39
size_t offset_mismatches
Lines that round-trip exactly whose ids match but offsets don't.
Definition tokenizer_parity.hpp:30
size_t id_mismatches
lines whose ids (with or without special tokens) differ
Definition tokenizer_parity.hpp:28
The tokenizer interface (TOK-1): a pipeline of normalizer, pre-tokenizer, model, post-processor and d...