The settings of a masked LM, from its Hugging Face config.json.
More...
#include <encoder_lm.hpp>
|
| std::string | architecture |
| | EsmForMaskedLM, EsmForDPLM, MDLM
|
| |
| std::string | model_type |
| | esm, mdlm
|
| |
| int64_t | vocab_size = 0 |
| |
| int64_t | hidden_size = 0 |
| |
| int64_t | intermediate_size = 0 |
| |
| int64_t | num_hidden_layers = 0 |
| |
| int64_t | num_attention_heads = 0 |
| |
| float | layer_norm_eps = 1e-12f |
| |
| float | rope_theta = 10000.0f |
| |
| int64_t | max_position_embeddings = 0 |
| |
| bool | token_dropout = false |
| | ESM's "mask dropout": masked tokens' embeddings are zeroed and the rest rescaled.
|
| |
| int64_t | mask_token_id = -1 |
| |
| int64_t | pad_token_id = -1 |
| |
| std::string | dtype |
| |
| std::vector< double > | rope_inverse_frequencies |
| | RoPE frequencies to use instead of rope_theta's, one per pair. LoadEncoderLM fills them from the checkpoint's inv_freq buffer: ESM-2's are stored rounded to fp16 (0.316162 rather than 0.316228), the model was trained with those, and transformers uses them, so recomputing them exactly shifts attention by about 2e-4.
|
| |
| EncoderLMHead | head = EncoderLMHead::Esm |
| |
| bool | attention_bias = true |
| | Biases on the attention's q, k, v and output projections (ESM's); MDLM has none.
|
| |
| ElementwiseOp | activation = ElementwiseOp::Gelu |
| | The MLP's activation: exact GELU for ESM, the tanh approximation for MDLM.
|
| |
The settings of a masked LM, from its Hugging Face config.json.
◆ activation
The MLP's activation: exact GELU for ESM, the tanh approximation for MDLM.
◆ architecture
| std::string pulsatrix::EncoderLMConfig::architecture |
EsmForMaskedLM, EsmForDPLM, MDLM
◆ attention_bias
| bool pulsatrix::EncoderLMConfig::attention_bias = true |
Biases on the attention's q, k, v and output projections (ESM's); MDLM has none.
◆ dtype
| std::string pulsatrix::EncoderLMConfig::dtype |
◆ head
◆ hidden_size
| int64_t pulsatrix::EncoderLMConfig::hidden_size = 0 |
◆ intermediate_size
| int64_t pulsatrix::EncoderLMConfig::intermediate_size = 0 |
◆ layer_norm_eps
| float pulsatrix::EncoderLMConfig::layer_norm_eps = 1e-12f |
◆ mask_token_id
| int64_t pulsatrix::EncoderLMConfig::mask_token_id = -1 |
◆ max_position_embeddings
| int64_t pulsatrix::EncoderLMConfig::max_position_embeddings = 0 |
◆ model_type
| std::string pulsatrix::EncoderLMConfig::model_type |
◆ num_attention_heads
| int64_t pulsatrix::EncoderLMConfig::num_attention_heads = 0 |
◆ num_hidden_layers
| int64_t pulsatrix::EncoderLMConfig::num_hidden_layers = 0 |
◆ pad_token_id
| int64_t pulsatrix::EncoderLMConfig::pad_token_id = -1 |
◆ rope_inverse_frequencies
| std::vector<double> pulsatrix::EncoderLMConfig::rope_inverse_frequencies |
RoPE frequencies to use instead of rope_theta's, one per pair. LoadEncoderLM fills them from the checkpoint's inv_freq buffer: ESM-2's are stored rounded to fp16 (0.316162 rather than 0.316228), the model was trained with those, and transformers uses them, so recomputing them exactly shifts attention by about 2e-4.
◆ rope_theta
| float pulsatrix::EncoderLMConfig::rope_theta = 10000.0f |
◆ token_dropout
| bool pulsatrix::EncoderLMConfig::token_dropout = false |
ESM's "mask dropout": masked tokens' embeddings are zeroed and the rest rescaled.
◆ vocab_size
| int64_t pulsatrix::EncoderLMConfig::vocab_size = 0 |
The documentation for this struct was generated from the following file: