pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::NormalizedString Class Reference

Text being normalized and pre-tokenized, remembering where each byte came from in the original input, so offsets survive changes such as NFC composition or byte-to-character mapping. More...

#include <text_tokenizer.hpp>

Classes

struct  Piece
 One piece of a rebuilt text: its new bytes, and the range of the current text they replace (an empty range inserts). More...
 

Public Member Functions

 NormalizedString ()=default
 
 NormalizedString (std::string text, size_t offset=0)
 text as it appears at byte offset of the original input, unchanged.
 
const std::string & text () const
 
bool empty () const
 
size_t size () const
 
Offset original (size_t begin, size_t end) const
 The original bytes that normalized bytes [begin, end) came from. An empty range gives an empty span where it sits.
 
NormalizedString slice (size_t begin, size_t end) const
 Normalized bytes [begin, end), keeping their origins.
 
void rebuild (const std::vector< Piece > &pieces)
 Replaces the text with pieces, in order. Every byte of a piece comes from the original span of its source range, which is how normalizers and byte-level mapping keep offsets.
 

Detailed Description

Text being normalized and pre-tokenized, remembering where each byte came from in the original input, so offsets survive changes such as NFC composition or byte-to-character mapping.

Constructor & Destructor Documentation

◆ NormalizedString() [1/2]

pulsatrix::NormalizedString::NormalizedString ( )
default

◆ NormalizedString() [2/2]

pulsatrix::NormalizedString::NormalizedString ( std::string  text,
size_t  offset = 0 
)
explicit

text as it appears at byte offset of the original input, unchanged.

Member Function Documentation

◆ empty()

bool pulsatrix::NormalizedString::empty ( ) const
inline

◆ original()

Offset pulsatrix::NormalizedString::original ( size_t  begin,
size_t  end 
) const

The original bytes that normalized bytes [begin, end) came from. An empty range gives an empty span where it sits.

Exceptions
std::out_of_rangeif the range doesn't fit the text.

◆ rebuild()

void pulsatrix::NormalizedString::rebuild ( const std::vector< Piece > &  pieces)

Replaces the text with pieces, in order. Every byte of a piece comes from the original span of its source range, which is how normalizers and byte-level mapping keep offsets.

Exceptions
std::out_of_rangeif a source range doesn't fit the current text.

◆ size()

size_t pulsatrix::NormalizedString::size ( ) const
inline

◆ slice()

NormalizedString pulsatrix::NormalizedString::slice ( size_t  begin,
size_t  end 
) const

Normalized bytes [begin, end), keeping their origins.

Exceptions
std::out_of_range.

◆ text()

const std::string & pulsatrix::NormalizedString::text ( ) const
inline

The documentation for this class was generated from the following file: