pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::TokenizerParityReport Struct Reference

How a tokenizer compares with the reference, field by field. More...

#include <tokenizer_parity.hpp>

Public Member Functions

bool passed () const
 

Public Attributes

size_t lines = 0
 
size_t id_mismatches = 0
 lines whose ids (with or without special tokens) differ
 
size_t offset_mismatches = 0
 Lines that round-trip exactly whose ids match but offsets don't.
 
size_t inexact_line_offset_differences = 0
 Lines that don't round-trip exactly whose offsets differ: counted, not failed. Such a line was changed by the normalizer (NFC) or lost characters the vocabulary lacks (dropped without an unknown token, as Hugging Face drops them). Hugging Face's offsets there are positional: a composed character gets only its first source character, characters after a canonical reordering get their neighbours' spans, and every token after a dropped character shifts left. pulsatrix tracks each byte's actual source.
 
size_t decode_mismatches = 0
 lines whose decoded text differs
 
std::vector< TokenizerMismatch > examples
 The first few mismatches, for diagnosis.
 

Detailed Description

How a tokenizer compares with the reference, field by field.

Member Function Documentation

◆ passed()

bool pulsatrix::TokenizerParityReport::passed ( ) const
inline

Member Data Documentation

◆ decode_mismatches

size_t pulsatrix::TokenizerParityReport::decode_mismatches = 0

lines whose decoded text differs

◆ examples

std::vector<TokenizerMismatch> pulsatrix::TokenizerParityReport::examples

The first few mismatches, for diagnosis.

◆ id_mismatches

size_t pulsatrix::TokenizerParityReport::id_mismatches = 0

lines whose ids (with or without special tokens) differ

◆ inexact_line_offset_differences

size_t pulsatrix::TokenizerParityReport::inexact_line_offset_differences = 0

Lines that don't round-trip exactly whose offsets differ: counted, not failed. Such a line was changed by the normalizer (NFC) or lost characters the vocabulary lacks (dropped without an unknown token, as Hugging Face drops them). Hugging Face's offsets there are positional: a composed character gets only its first source character, characters after a canonical reordering get their neighbours' spans, and every token after a dropped character shifts left. pulsatrix tracks each byte's actual source.

◆ lines

size_t pulsatrix::TokenizerParityReport::lines = 0

◆ offset_mismatches

size_t pulsatrix::TokenizerParityReport::offset_mismatches = 0

Lines that round-trip exactly whose ids match but offsets don't.


The documentation for this struct was generated from the following file: