pulsatrix 1.1.0
Loading...
Searching...
No Matches
tokenizer_parity.hpp File Reference

Compares a TextTokenizer with Hugging Face tokenizers on a reference corpus (TOK-2), as written by tools/tokenizers/make_tokenizer_reference.py. More...

#include <cstddef>
#include <string>
#include <vector>
#include "pulsatrix/text_tokenizer.hpp"
Include dependency graph for tokenizer_parity.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::TokenizerMismatch
 The first difference on one line of the corpus. More...
 
struct  pulsatrix::TokenizerParityReport
 How a tokenizer compares with the reference, field by field. More...
 

Namespaces

namespace  pulsatrix
 

Functions

TokenizerParityReport pulsatrix::CompareToTokenizerReference (const TextTokenizer &tokenizer, const std::string &path, size_t max_examples=5)
 Encodes and decodes every line of a reference file (JSON lines with text, ids, plain_ids, offsets in characters, decoded, decoded_skip, and normalized: whether the normalizer changed the line) and counts the lines that differ. A line round-trips exactly when the normalizer leaves it alone and decoding its plain ids gives it back; offsets are compared exactly on those lines.
 
Offset pulsatrix::ByteToCharOffset (std::string_view text, Offset bytes)
 A byte offset range of text as a character (code point) range, widened to whole characters, which is how Hugging Face reports offsets.
 

Detailed Description

Compares a TextTokenizer with Hugging Face tokenizers on a reference corpus (TOK-2), as written by tools/tokenizers/make_tokenizer_reference.py.