Text being normalized and pre-tokenized, remembering where each byte came from in the original input, so offsets survive changes such as NFC composition or byte-to-character mapping.
More...
#include <text_tokenizer.hpp>
|
| struct | Piece |
| | One piece of a rebuilt text: its new bytes, and the range of the current text they replace (an empty range inserts). More...
|
| |
|
| | NormalizedString ()=default |
| |
| | NormalizedString (std::string text, size_t offset=0) |
| | text as it appears at byte offset of the original input, unchanged.
|
| |
| const std::string & | text () const |
| |
| bool | empty () const |
| |
| size_t | size () const |
| |
| Offset | original (size_t begin, size_t end) const |
| | The original bytes that normalized bytes [begin, end) came from. An empty range gives an empty span where it sits.
|
| |
| NormalizedString | slice (size_t begin, size_t end) const |
| | Normalized bytes [begin, end), keeping their origins.
|
| |
| void | rebuild (const std::vector< Piece > &pieces) |
| | Replaces the text with pieces, in order. Every byte of a piece comes from the original span of its source range, which is how normalizers and byte-level mapping keep offsets.
|
| |
Text being normalized and pre-tokenized, remembering where each byte came from in the original input, so offsets survive changes such as NFC composition or byte-to-character mapping.
◆ NormalizedString() [1/2]
| pulsatrix::NormalizedString::NormalizedString |
( |
| ) |
|
|
default |
◆ NormalizedString() [2/2]
| pulsatrix::NormalizedString::NormalizedString |
( |
std::string |
text, |
|
|
size_t |
offset = 0 |
|
) |
| |
|
explicit |
text as it appears at byte offset of the original input, unchanged.
◆ empty()
| bool pulsatrix::NormalizedString::empty |
( |
| ) |
const |
|
inline |
◆ original()
| Offset pulsatrix::NormalizedString::original |
( |
size_t |
begin, |
|
|
size_t |
end |
|
) |
| const |
The original bytes that normalized bytes [begin, end) came from. An empty range gives an empty span where it sits.
- Exceptions
-
| std::out_of_range | if the range doesn't fit the text. |
◆ rebuild()
| void pulsatrix::NormalizedString::rebuild |
( |
const std::vector< Piece > & |
pieces | ) |
|
Replaces the text with pieces, in order. Every byte of a piece comes from the original span of its source range, which is how normalizers and byte-level mapping keep offsets.
- Exceptions
-
| std::out_of_range | if a source range doesn't fit the current text. |
◆ size()
| size_t pulsatrix::NormalizedString::size |
( |
| ) |
const |
|
inline |
◆ slice()
| NormalizedString pulsatrix::NormalizedString::slice |
( |
size_t |
begin, |
|
|
size_t |
end |
|
) |
| const |
Normalized bytes [begin, end), keeping their origins.
- Exceptions
-
◆ text()
| const std::string & pulsatrix::NormalizedString::text |
( |
| ) |
const |
|
inline |
The documentation for this class was generated from the following file: