pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::EncoderLMConfig Struct Reference

The settings of a masked LM, from its Hugging Face config.json. More...

#include <encoder_lm.hpp>

Public Attributes

std::string architecture
 EsmForMaskedLM, EsmForDPLM, MDLM
 
std::string model_type
 esm, mdlm
 
int64_t vocab_size = 0
 
int64_t hidden_size = 0
 
int64_t intermediate_size = 0
 
int64_t num_hidden_layers = 0
 
int64_t num_attention_heads = 0
 
float layer_norm_eps = 1e-12f
 
float rope_theta = 10000.0f
 
int64_t max_position_embeddings = 0
 
bool token_dropout = false
 ESM's "mask dropout": masked tokens' embeddings are zeroed and the rest rescaled.
 
int64_t mask_token_id = -1
 
int64_t pad_token_id = -1
 
std::string dtype
 
std::vector< double > rope_inverse_frequencies
 RoPE frequencies to use instead of rope_theta's, one per pair. LoadEncoderLM fills them from the checkpoint's inv_freq buffer: ESM-2's are stored rounded to fp16 (0.316162 rather than 0.316228), the model was trained with those, and transformers uses them, so recomputing them exactly shifts attention by about 2e-4.
 
EncoderLMHead head = EncoderLMHead::Esm
 
bool attention_bias = true
 Biases on the attention's q, k, v and output projections (ESM's); MDLM has none.
 
ElementwiseOp activation = ElementwiseOp::Gelu
 The MLP's activation: exact GELU for ESM, the tanh approximation for MDLM.
 

Detailed Description

The settings of a masked LM, from its Hugging Face config.json.

Member Data Documentation

◆ activation

ElementwiseOp pulsatrix::EncoderLMConfig::activation = ElementwiseOp::Gelu

The MLP's activation: exact GELU for ESM, the tanh approximation for MDLM.

◆ architecture

std::string pulsatrix::EncoderLMConfig::architecture

EsmForMaskedLM, EsmForDPLM, MDLM

◆ attention_bias

bool pulsatrix::EncoderLMConfig::attention_bias = true

Biases on the attention's q, k, v and output projections (ESM's); MDLM has none.

◆ dtype

std::string pulsatrix::EncoderLMConfig::dtype

◆ head

EncoderLMHead pulsatrix::EncoderLMConfig::head = EncoderLMHead::Esm

◆ hidden_size

int64_t pulsatrix::EncoderLMConfig::hidden_size = 0

◆ intermediate_size

int64_t pulsatrix::EncoderLMConfig::intermediate_size = 0

◆ layer_norm_eps

float pulsatrix::EncoderLMConfig::layer_norm_eps = 1e-12f

◆ mask_token_id

int64_t pulsatrix::EncoderLMConfig::mask_token_id = -1

◆ max_position_embeddings

int64_t pulsatrix::EncoderLMConfig::max_position_embeddings = 0

◆ model_type

std::string pulsatrix::EncoderLMConfig::model_type

esm, mdlm

◆ num_attention_heads

int64_t pulsatrix::EncoderLMConfig::num_attention_heads = 0

◆ num_hidden_layers

int64_t pulsatrix::EncoderLMConfig::num_hidden_layers = 0

◆ pad_token_id

int64_t pulsatrix::EncoderLMConfig::pad_token_id = -1

◆ rope_inverse_frequencies

std::vector<double> pulsatrix::EncoderLMConfig::rope_inverse_frequencies

RoPE frequencies to use instead of rope_theta's, one per pair. LoadEncoderLM fills them from the checkpoint's inv_freq buffer: ESM-2's are stored rounded to fp16 (0.316162 rather than 0.316228), the model was trained with those, and transformers uses them, so recomputing them exactly shifts attention by about 2e-4.

◆ rope_theta

float pulsatrix::EncoderLMConfig::rope_theta = 10000.0f

◆ token_dropout

bool pulsatrix::EncoderLMConfig::token_dropout = false

ESM's "mask dropout": masked tokens' embeddings are zeroed and the rest rescaled.

◆ vocab_size

int64_t pulsatrix::EncoderLMConfig::vocab_size = 0

The documentation for this struct was generated from the following file: