pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::TokenizerMismatch Struct Reference

The first difference on one line of the corpus. More...

#include <tokenizer_parity.hpp>

Public Attributes

size_t line = 0
 
std::string field
 "ids", "plain_ids", "offsets", "decoded" or "decoded_skip".
 
std::string text
 
std::string detail
 

Detailed Description

The first difference on one line of the corpus.

Member Data Documentation

◆ detail

std::string pulsatrix::TokenizerMismatch::detail

◆ field

std::string pulsatrix::TokenizerMismatch::field

"ids", "plain_ids", "offsets", "decoded" or "decoded_skip".

◆ line

size_t pulsatrix::TokenizerMismatch::line = 0

◆ text

std::string pulsatrix::TokenizerMismatch::text

The documentation for this struct was generated from the following file: