Token<->index lookup with a reserved <unk> fallback, plus a frequency-ranked builder. More...

#include <cstdint>
#include <string>
#include <unordered_map>
#include <vector>

Include dependency graph for vocabulary.hpp:

This graph shows which files directly or indirectly include this file:

Classes
class	pulsatrix::Vocabulary
	Token<->index lookup table. Index 0 is always the reserved "<unk>" token – guaranteed by construction, not caller convention: the constructor takes the ranked list of real tokens and prepends "<unk>" itself. More...

Functions
Vocabulary	pulsatrix::BuildVocabulary (const std::vector< std::vector< std::string > > &tokenized_corpus, int64_t max_vocab_size=-1)
	Builds a Vocabulary from a tokenized corpus, ranked by descending token frequency (ties broken by first-seen order, for determinism).

Detailed Description

Token<->index lookup with a reserved <unk> fallback, plus a frequency-ranked builder.

Classes