Turns sequences into masked-LM batches as ESM-2 was trained: <cls> + residues + <eos>, cropped to a random window when too long, then each residue picked with probability 0.15; of those, 80% become <mask>, 10% a random amino acid (uniform over the 20 standard ones) and 10% stay. <cls>, <eos> and padding are never picked. Draws come from a seeded portable generator, so a seed gives the same batches on every platform.
More...
#include <protein_training.hpp>
Turns sequences into masked-LM batches as ESM-2 was trained: <cls> + residues + <eos>, cropped to a random window when too long, then each residue picked with probability 0.15; of those, 80% become <mask>, 10% a random amino acid (uniform over the 20 standard ones) and 10% stay. <cls>, <eos> and padding are never picked. Draws come from a seeded portable generator, so a seed gives the same batches on every platform.
- Note
- Token dropout (ESM-2 zeroes
<mask> embeddings and rescales the rest) is EncoderLM's forward(), in training as in inference.
◆ MaskedLMCollator()
- Exceptions
-
| std::invalid_argument | if the tokenizer lacks a special token or an amino acid, or the options are out of range (probabilities outside [0, 1], fractions summing over 1, max_tokens < 3). |
◆ collate()
| MaskedLMBatch pulsatrix::MaskedLMCollator::collate |
( |
const std::vector< std::string > & |
sequences | ) |
|
- Exceptions
-
| std::invalid_argument | for no sequences, an empty one, or a character the tokenizer doesn't know. |
◆ tokens()
| std::vector< int64_t > pulsatrix::MaskedLMCollator::tokens |
( |
const std::string & |
sequence | ) |
|
<cls> + sequence (cropped as collate() would) + <eos>, unmasked.
The documentation for this class was generated from the following file: