pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::MaskedLMCollator Class Reference

Turns sequences into masked-LM batches as ESM-2 was trained: <cls> + residues + <eos>, cropped to a random window when too long, then each residue picked with probability 0.15; of those, 80% become <mask>, 10% a random amino acid (uniform over the 20 standard ones) and 10% stay. <cls>, <eos> and padding are never picked. Draws come from a seeded portable generator, so a seed gives the same batches on every platform. More...

#include <protein_training.hpp>

Public Member Functions

 MaskedLMCollator (const TextTokenizer &tokenizer, DeviceBackend *backend, MaskedLMOptions options={}, uint64_t seed=0)
 
MaskedLMBatch collate (const std::vector< std::string > &sequences)
 
std::vector< int64_t > tokens (const std::string &sequence)
 <cls> + sequence (cropped as collate() would) + <eos>, unmasked.
 

Detailed Description

Turns sequences into masked-LM batches as ESM-2 was trained: <cls> + residues + <eos>, cropped to a random window when too long, then each residue picked with probability 0.15; of those, 80% become <mask>, 10% a random amino acid (uniform over the 20 standard ones) and 10% stay. <cls>, <eos> and padding are never picked. Draws come from a seeded portable generator, so a seed gives the same batches on every platform.

Note
Token dropout (ESM-2 zeroes <mask> embeddings and rescales the rest) is EncoderLM's forward(), in training as in inference.

Constructor & Destructor Documentation

◆ MaskedLMCollator()

pulsatrix::MaskedLMCollator::MaskedLMCollator ( const TextTokenizer &  tokenizer,
DeviceBackend *  backend,
MaskedLMOptions  options = {},
uint64_t  seed = 0 
)
Exceptions
std::invalid_argumentif the tokenizer lacks a special token or an amino acid, or the options are out of range (probabilities outside [0, 1], fractions summing over 1, max_tokens < 3).

Member Function Documentation

◆ collate()

MaskedLMBatch pulsatrix::MaskedLMCollator::collate ( const std::vector< std::string > &  sequences)
Exceptions
std::invalid_argumentfor no sequences, an empty one, or a character the tokenizer doesn't know.

◆ tokens()

std::vector< int64_t > pulsatrix::MaskedLMCollator::tokens ( const std::string &  sequence)

<cls> + sequence (cropped as collate() would) + <eos>, unmasked.


The documentation for this class was generated from the following file: