pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::WordPunctuationSplit Class Reference

The split Tokenizer::Tokenize makes: runs of ASCII letters, digits and apostrophes are words, ASCII whitespace separates, and every other character is a token of its own. More...

#include <tokenizer_components.hpp>

Inheritance diagram for pulsatrix::WordPunctuationSplit:
Collaboration diagram for pulsatrix::WordPunctuationSplit:

Public Member Functions

void pre_tokenize (std::vector< NormalizedString > &splits) const override
 Replaces splits with their further-split (or transformed) pieces, in order.
 
- Public Member Functions inherited from pulsatrix::PreTokenizer
virtual ~PreTokenizer ()=default
 

Detailed Description

The split Tokenizer::Tokenize makes: runs of ASCII letters, digits and apostrophes are words, ASCII whitespace separates, and every other character is a token of its own.

Note
Tokenizer::Tokenize works byte by byte and splits a non-ASCII character into its bytes; this keeps the character whole. On ASCII text the two agree.

Member Function Documentation

◆ pre_tokenize()

void pulsatrix::WordPunctuationSplit::pre_tokenize ( std::vector< NormalizedString > &  splits) const
overridevirtual

Replaces splits with their further-split (or transformed) pieces, in order.

Implements pulsatrix::PreTokenizer.


The documentation for this class was generated from the following file: