|
pulsatrix 1.1.0
|
What a protein language model's representations encode (PLM-8): per-residue concepts from Swiss-Prot annotations, per-layer linear probes with control tasks, and sparse autoencoder features matched to the concepts, as InterPLM does. More...
#include <cstdint>#include <string>#include <string_view>#include <utility>#include <vector>#include "pulsatrix/encoder_lm.hpp"#include "pulsatrix/featurizer.hpp"#include "pulsatrix/probe_controls.hpp"#include "pulsatrix/text_tokenizer.hpp"
Go to the source code of this file.
Classes | |
| struct | pulsatrix::ProteinFeature |
| One annotated stretch, numbered from 1, inclusive, as UniProt numbers it. More... | |
| struct | pulsatrix::AnnotatedProtein |
| struct | pulsatrix::ResidueEmbeddings |
Per-residue representations at some layers: for each layer, residues x hidden values, the proteins' residues one after another. More... | |
| struct | pulsatrix::SparseCodes |
| Sparse codes: for each residue, its active features and their values. More... | |
| struct | pulsatrix::ConceptMatch |
| The feature that best matches one concept. More... | |
Namespaces | |
| namespace | pulsatrix |
Functions | |
| std::vector< AnnotatedProtein > | pulsatrix::ParseAnnotatedProteins (std::string_view jsonl) |
Reads JSON lines, one protein per line: {"accession", "sequence", "features": [{"type", "start", "end"}]} (tools/plm/fetch_swissprot_annotations.py writes them). Blank lines are skipped. | |
| std::vector< AnnotatedProtein > | pulsatrix::ReadAnnotatedProteins (const std::string &path) |
| ParseAnnotatedProteins on a file. | |
| std::vector< int > | pulsatrix::ConceptLabels (const AnnotatedProtein &protein, const std::string &type) |
1 for each residue a feature of type covers, 0 elsewhere. A disulfide bond marks only its two cysteines (UniProt gives them as start and end), not the stretch between. | |
| std::vector< int > | pulsatrix::SecondaryStructureLabels (const AnnotatedProtein &protein) |
Three-state secondary structure per residue: 0 helix (Helix), 1 strand (Beta strand), 2 anything else (turns and coil). | |
| std::vector< std::pair< std::string, int64_t > > | pulsatrix::ConceptCounts (const std::vector< AnnotatedProtein > &proteins) |
Each feature type, with the residues it covers over proteins, most first. | |
| ResidueEmbeddings | pulsatrix::EmbedResidues (EncoderLM &model, const TextTokenizer &tokenizer, DeviceBackend *backend, const std::vector< std::string > &sequences, const std::vector< int64_t > &layers, int64_t max_tokens=1024) |
Runs sequences through the model one at a time and keeps each residue's representation (not <cls> or <eos>) at layers: 0 is the embeddings, i the output of layer i, and num_layers() the final representation, after the final LayerNorm, as ESM embeddings are usually taken. | |
| std::vector< float > | pulsatrix::SequenceWindowFeatures (const std::vector< std::string > &sequences, int64_t window) |
The control a residue probe has to beat: each residue's local sequence, one-hot over the 20 amino acids at every position within window of it (20 * (2 window + 1) values, zeros past the ends and for other letters). A linear probe that reads a concept from a layer no better than from this hasn't shown the model adds anything beyond nearby sequence. | |
| SparseCodes | pulsatrix::EncodeSparse (Featurizer &featurizer, const std::vector< float > &rows, DeviceBackend *backend, int64_t batch=4096) |
featurizer's codes for every residue in rows (residues x input_dim), encoded in batches of batch and kept where above zero. | |
| SparseCodes | pulsatrix::NeuronCodes (const std::vector< float > &rows, int64_t dim) |
Dense values as codes, each dimension d twice: as feature 2d (its positive part) and 2d + 1 (its negative part, flipped), so single neurons can be matched to concepts the same way as features. | |
| ConceptMatch | pulsatrix::MatchConcept (const SparseCodes &codes, const std::vector< int > &labels, const std::string &concept_name, const std::vector< double > &thresholds={0.0, 0.15, 0.5, 0.6, 0.8}) |
InterPLM's matching: a feature predicts the concept on a residue when its value exceeds a threshold, a share of its largest value (thresholds), and is scored by F1 over the residues. Returns the best feature and threshold. | |
What a protein language model's representations encode (PLM-8): per-residue concepts from Swiss-Prot annotations, per-layer linear probes with control tasks, and sparse autoencoder features matched to the concepts, as InterPLM does.
Probes and features answer different questions. A probe asks whether a concept can be read out of a layer linearly; a high score can also mean the probe learned the task from residue identity alone, so it comes with a local-sequence control and a randomly initialized model. An SAE asks which directions the model uses unprompted; a feature matching a concept is a lead, not proof.