tokenizer.hpp File Reference

Minimal deterministic whitespace/punctuation word-level tokenizer. More...

#include <string>
#include <vector>

Include dependency graph for tokenizer.hpp:

Go to the source code of this file.

Classes
class	pulsatrix::Tokenizer
	Splits text into lowercase word/punctuation tokens – pulsatrix's first text primitive (campaign_exai_dl_library_data_pipeline, Phase 3, Decision Point 6: a minimal whitespace/punctuation tokenizer, not BPE – training a real subword-merge algorithm is a project-sized undertaking on its own). More...

Namespaces
namespace	pulsatrix

Detailed Description

Minimal deterministic whitespace/punctuation word-level tokenizer.