|
pulsatrix 1.1.0
|
Training protein language models (PLM-7): ESM-2's masked-LM data pipeline (masking, cropping, cluster-weighted sampling), its initialization, a training step and evaluation, and heads for fine-tuning on per-protein and per-residue labels. More...
#include <cstdint>#include <string>#include <vector>#include "pulsatrix/encoder_lm.hpp"#include "pulsatrix/linear_module.hpp"#include "pulsatrix/text_tokenizer.hpp"#include "pulsatrix/token_cross_entropy_loss.hpp"
Go to the source code of this file.
Classes | |
| struct | pulsatrix::MaskedLMOptions |
| ESM-2's masking (Lin et al. 2023; BERT's 80/10/10). More... | |
| struct | pulsatrix::MaskedLMBatch |
| One collated batch, on the collator's backend. More... | |
| class | pulsatrix::MaskedLMCollator |
Turns sequences into masked-LM batches as ESM-2 was trained: <cls> + residues + <eos>, cropped to a random window when too long, then each residue picked with probability 0.15; of those, 80% become <mask>, 10% a random amino acid (uniform over the 20 standard ones) and 10% stay. <cls>, <eos> and padding are never picked. Draws come from a seeded portable generator, so a seed gives the same batches on every platform. More... | |
| class | pulsatrix::ClusterSampler |
| Cluster-weighted sampling, as ESM-2 drew UniRef50: a cluster uniformly, then one of its members uniformly, so large families don't dominate. With every sequence its own cluster, it is uniform sampling with replacement. More... | |
| struct | pulsatrix::MaskedLMEvaluation |
| Masked-LM quality on held-out sequences. More... | |
| class | pulsatrix::SequenceHead |
A head on an encoder's representations (last_hidden_state), for fine-tuning or probing. Pooling::Mean averages each sequence's residues (not <cls>, <eos> or padding) and applies a linear layer: (N, L, hidden) to (N, outputs), for per-protein labels. Pooling::PerResidue applies it to every token: (N, L, hidden) to (N, L, outputs), for per-residue labels (use -100 targets at <cls>, <eos> and padding). More... | |
Namespaces | |
| namespace | pulsatrix |
Functions | |
| void | pulsatrix::InitializeEsm (EncoderLM &model, uint64_t seed, float std=0.02f) |
Initializes a model for training from scratch as transformers' EsmPreTrainedModel does: linear weights and the embedding table from N(0, std²) (the <pad> row zeroed), biases 0, LayerNorm gains 1 and offsets 0. Deterministic for a seed. | |
| float | pulsatrix::MaskedLMForwardBackward (EncoderLM &model, const MaskedLMBatch &batch, TokenCrossEntropyLoss &loss, float normalizer=0.0f) |
One masked-LM forward and backward: sets the batch's padding mask, runs the model, and accumulates the gradient of the cross-entropy over the picked tokens, divided by normalizer (0: the batch's own num_targets, i.e. the mean). Gradients add to what's there; zero them between optimizer steps. | |
| MaskedLMEvaluation | pulsatrix::EvaluateMaskedLM (EncoderLM &model, const TextTokenizer &tokenizer, DeviceBackend *backend, const std::vector< std::string > &sequences, int64_t batch_size=8, uint64_t seed=0, MaskedLMOptions options={}) |
Masks sequences with a collator seeded by seed (so every model sees the same masks) and measures the loss over the picked tokens, in batches of batch_size. The model keeps no activations while it runs. | |
| Tensor | pulsatrix::ResidueMask (const Tensor &ids, const Tensor *keep, const TextTokenizer &tokenizer, DeviceBackend *backend) |
The residue mask for SequenceHead::Pooling::Mean from token ids and a padding mask (or none): 1 for every real token that isn't <cls> or <eos>. | |
Training protein language models (PLM-7): ESM-2's masked-LM data pipeline (masking, cropping, cluster-weighted sampling), its initialization, a training step and evaluation, and heads for fine-tuning on per-protein and per-residue labels.