49[[nodiscard]] std::vector<FastaRecord>
ParseFasta(std::string_view text);
51[[nodiscard]] std::vector<FastaRecord>
ReadFasta(
const std::string& path);
53[[nodiscard]] std::string
WriteFasta(
const std::vector<FastaRecord>& records,
size_t width = 60);
A tokenizer: added tokens are split out of the raw text first (longest match, earliest first); each r...
Definition text_tokenizer.hpp:166
Definition acquisition_functions.hpp:16
std::string WriteFasta(const std::vector< FastaRecord > &records, size_t width=60)
FASTA text for records, sequences wrapped at width residues (0: one line).
std::vector< FastaRecord > ParseFasta(std::string_view text)
Parses FASTA text: records start with >, sequences may span lines, and blank lines and ; comment line...
TextTokenizer MakeEsmTokenizer(const std::vector< std::string > &vocab)
ESM's tokenizer (Hugging Face's EsmTokenizer) over a vocabulary listed one token per line,...
TextTokenizer LoadEsmTokenizer(const std::string &vocab_path)
MakeEsmTokenizer on a vocab.txt.
std::vector< FastaRecord > ReadFasta(const std::string &path)
ParseFasta on a file.
One FASTA record.
Definition protein_sequences.hpp:34
std::string description
The rest of the header line, after the first run of whitespace.
Definition protein_sequences.hpp:38
std::string id
The header's first word, without the >.
Definition protein_sequences.hpp:36
std::string sequence
The sequence lines joined, with whitespace removed. Case and characters are kept as written,...
Definition protein_sequences.hpp:41
The tokenizer interface (TOK-1): a pipeline of normalizer, pre-tokenizer, model, post-processor and d...