pulsatrix 1.1.0
Loading...
Searching...
No Matches
protein_training.hpp File Reference

Training protein language models (PLM-7): ESM-2's masked-LM data pipeline (masking, cropping, cluster-weighted sampling), its initialization, a training step and evaluation, and heads for fine-tuning on per-protein and per-residue labels. More...

#include <cstdint>
#include <string>
#include <vector>
#include "pulsatrix/encoder_lm.hpp"
#include "pulsatrix/linear_module.hpp"
#include "pulsatrix/text_tokenizer.hpp"
#include "pulsatrix/token_cross_entropy_loss.hpp"
Include dependency graph for protein_training.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::MaskedLMOptions
 ESM-2's masking (Lin et al. 2023; BERT's 80/10/10). More...
 
struct  pulsatrix::MaskedLMBatch
 One collated batch, on the collator's backend. More...
 
class  pulsatrix::MaskedLMCollator
 Turns sequences into masked-LM batches as ESM-2 was trained: <cls> + residues + <eos>, cropped to a random window when too long, then each residue picked with probability 0.15; of those, 80% become <mask>, 10% a random amino acid (uniform over the 20 standard ones) and 10% stay. <cls>, <eos> and padding are never picked. Draws come from a seeded portable generator, so a seed gives the same batches on every platform. More...
 
class  pulsatrix::ClusterSampler
 Cluster-weighted sampling, as ESM-2 drew UniRef50: a cluster uniformly, then one of its members uniformly, so large families don't dominate. With every sequence its own cluster, it is uniform sampling with replacement. More...
 
struct  pulsatrix::MaskedLMEvaluation
 Masked-LM quality on held-out sequences. More...
 
class  pulsatrix::SequenceHead
 A head on an encoder's representations (last_hidden_state), for fine-tuning or probing. Pooling::Mean averages each sequence's residues (not <cls>, <eos> or padding) and applies a linear layer: (N, L, hidden) to (N, outputs), for per-protein labels. Pooling::PerResidue applies it to every token: (N, L, hidden) to (N, L, outputs), for per-residue labels (use -100 targets at <cls>, <eos> and padding). More...
 

Namespaces

namespace  pulsatrix
 

Functions

void pulsatrix::InitializeEsm (EncoderLM &model, uint64_t seed, float std=0.02f)
 Initializes a model for training from scratch as transformers' EsmPreTrainedModel does: linear weights and the embedding table from N(0, std²) (the <pad> row zeroed), biases 0, LayerNorm gains 1 and offsets 0. Deterministic for a seed.
 
float pulsatrix::MaskedLMForwardBackward (EncoderLM &model, const MaskedLMBatch &batch, TokenCrossEntropyLoss &loss, float normalizer=0.0f)
 One masked-LM forward and backward: sets the batch's padding mask, runs the model, and accumulates the gradient of the cross-entropy over the picked tokens, divided by normalizer (0: the batch's own num_targets, i.e. the mean). Gradients add to what's there; zero them between optimizer steps.
 
MaskedLMEvaluation pulsatrix::EvaluateMaskedLM (EncoderLM &model, const TextTokenizer &tokenizer, DeviceBackend *backend, const std::vector< std::string > &sequences, int64_t batch_size=8, uint64_t seed=0, MaskedLMOptions options={})
 Masks sequences with a collator seeded by seed (so every model sees the same masks) and measures the loss over the picked tokens, in batches of batch_size. The model keeps no activations while it runs.
 
Tensor pulsatrix::ResidueMask (const Tensor &ids, const Tensor *keep, const TextTokenizer &tokenizer, DeviceBackend *backend)
 The residue mask for SequenceHead::Pooling::Mean from token ids and a padding mask (or none): 1 for every real token that isn't <cls> or <eos>.
 

Detailed Description

Training protein language models (PLM-7): ESM-2's masked-LM data pipeline (masking, cropping, cluster-weighted sampling), its initialization, a training step and evaluation, and heads for fine-tuning on per-protein and per-residue labels.