How a tokenizer compares with the reference, field by field.
More...
#include <tokenizer_parity.hpp>
|
| size_t | lines = 0 |
| |
| size_t | id_mismatches = 0 |
| | lines whose ids (with or without special tokens) differ
|
| |
| size_t | offset_mismatches = 0 |
| | Lines that round-trip exactly whose ids match but offsets don't.
|
| |
| size_t | inexact_line_offset_differences = 0 |
| | Lines that don't round-trip exactly whose offsets differ: counted, not failed. Such a line was changed by the normalizer (NFC) or lost characters the vocabulary lacks (dropped without an unknown token, as Hugging Face drops them). Hugging Face's offsets there are positional: a composed character gets only its first source character, characters after a canonical reordering get their neighbours' spans, and every token after a dropped character shifts left. pulsatrix tracks each byte's actual source.
|
| |
| size_t | decode_mismatches = 0 |
| | lines whose decoded text differs
|
| |
| std::vector< TokenizerMismatch > | examples |
| | The first few mismatches, for diagnosis.
|
| |
How a tokenizer compares with the reference, field by field.
◆ passed()
| bool pulsatrix::TokenizerParityReport::passed |
( |
| ) |
const |
|
inline |
◆ decode_mismatches
| size_t pulsatrix::TokenizerParityReport::decode_mismatches = 0 |
lines whose decoded text differs
◆ examples
The first few mismatches, for diagnosis.
◆ id_mismatches
| size_t pulsatrix::TokenizerParityReport::id_mismatches = 0 |
lines whose ids (with or without special tokens) differ
◆ inexact_line_offset_differences
| size_t pulsatrix::TokenizerParityReport::inexact_line_offset_differences = 0 |
Lines that don't round-trip exactly whose offsets differ: counted, not failed. Such a line was changed by the normalizer (NFC) or lost characters the vocabulary lacks (dropped without an unknown token, as Hugging Face drops them). Hugging Face's offsets there are positional: a composed character gets only its first source character, characters after a canonical reordering get their neighbours' spans, and every token after a dropped character shifts left. pulsatrix tracks each byte's actual source.
◆ lines
| size_t pulsatrix::TokenizerParityReport::lines = 0 |
◆ offset_mismatches
| size_t pulsatrix::TokenizerParityReport::offset_mismatches = 0 |
Lines that round-trip exactly whose ids match but offsets don't.
The documentation for this struct was generated from the following file: