pulsatrix 1.1.0
Loading...
Searching...
No Matches
protein_concepts.hpp File Reference

What a protein language model's representations encode (PLM-8): per-residue concepts from Swiss-Prot annotations, per-layer linear probes with control tasks, and sparse autoencoder features matched to the concepts, as InterPLM does. More...

#include <cstdint>
#include <string>
#include <string_view>
#include <utility>
#include <vector>
#include "pulsatrix/encoder_lm.hpp"
#include "pulsatrix/featurizer.hpp"
#include "pulsatrix/probe_controls.hpp"
#include "pulsatrix/text_tokenizer.hpp"
Include dependency graph for protein_concepts.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::ProteinFeature
 One annotated stretch, numbered from 1, inclusive, as UniProt numbers it. More...
 
struct  pulsatrix::AnnotatedProtein
 
struct  pulsatrix::ResidueEmbeddings
 Per-residue representations at some layers: for each layer, residues x hidden values, the proteins' residues one after another. More...
 
struct  pulsatrix::SparseCodes
 Sparse codes: for each residue, its active features and their values. More...
 
struct  pulsatrix::ConceptMatch
 The feature that best matches one concept. More...
 

Namespaces

namespace  pulsatrix
 

Functions

std::vector< AnnotatedProtein > pulsatrix::ParseAnnotatedProteins (std::string_view jsonl)
 Reads JSON lines, one protein per line: {"accession", "sequence", "features": [{"type", "start", "end"}]} (tools/plm/fetch_swissprot_annotations.py writes them). Blank lines are skipped.
 
std::vector< AnnotatedProtein > pulsatrix::ReadAnnotatedProteins (const std::string &path)
 ParseAnnotatedProteins on a file.
 
std::vector< int > pulsatrix::ConceptLabels (const AnnotatedProtein &protein, const std::string &type)
 1 for each residue a feature of type covers, 0 elsewhere. A disulfide bond marks only its two cysteines (UniProt gives them as start and end), not the stretch between.
 
std::vector< int > pulsatrix::SecondaryStructureLabels (const AnnotatedProtein &protein)
 Three-state secondary structure per residue: 0 helix (Helix), 1 strand (Beta strand), 2 anything else (turns and coil).
 
std::vector< std::pair< std::string, int64_t > > pulsatrix::ConceptCounts (const std::vector< AnnotatedProtein > &proteins)
 Each feature type, with the residues it covers over proteins, most first.
 
ResidueEmbeddings pulsatrix::EmbedResidues (EncoderLM &model, const TextTokenizer &tokenizer, DeviceBackend *backend, const std::vector< std::string > &sequences, const std::vector< int64_t > &layers, int64_t max_tokens=1024)
 Runs sequences through the model one at a time and keeps each residue's representation (not <cls> or <eos>) at layers: 0 is the embeddings, i the output of layer i, and num_layers() the final representation, after the final LayerNorm, as ESM embeddings are usually taken.
 
std::vector< float > pulsatrix::SequenceWindowFeatures (const std::vector< std::string > &sequences, int64_t window)
 The control a residue probe has to beat: each residue's local sequence, one-hot over the 20 amino acids at every position within window of it (20 * (2 window + 1) values, zeros past the ends and for other letters). A linear probe that reads a concept from a layer no better than from this hasn't shown the model adds anything beyond nearby sequence.
 
SparseCodes pulsatrix::EncodeSparse (Featurizer &featurizer, const std::vector< float > &rows, DeviceBackend *backend, int64_t batch=4096)
 featurizer's codes for every residue in rows (residues x input_dim), encoded in batches of batch and kept where above zero.
 
SparseCodes pulsatrix::NeuronCodes (const std::vector< float > &rows, int64_t dim)
 Dense values as codes, each dimension d twice: as feature 2d (its positive part) and 2d + 1 (its negative part, flipped), so single neurons can be matched to concepts the same way as features.
 
ConceptMatch pulsatrix::MatchConcept (const SparseCodes &codes, const std::vector< int > &labels, const std::string &concept_name, const std::vector< double > &thresholds={0.0, 0.15, 0.5, 0.6, 0.8})
 InterPLM's matching: a feature predicts the concept on a residue when its value exceeds a threshold, a share of its largest value (thresholds), and is scored by F1 over the residues. Returns the best feature and threshold.
 

Detailed Description

What a protein language model's representations encode (PLM-8): per-residue concepts from Swiss-Prot annotations, per-layer linear probes with control tasks, and sparse autoencoder features matched to the concepts, as InterPLM does.

Probes and features answer different questions. A probe asks whether a concept can be read out of a layer linearly; a high score can also mean the probe learned the task from residue identity alone, so it comes with a local-sequence control and a randomly initialized model. An SAE asks which directions the model uses unprompted; a feature matching a concept is a lead, not proof.