|
pulsatrix 1.1.0
|
Everything about an EncoderBlock beyond its attention. The defaults are ESM-2's. More...
#include <encoder_block.hpp>
Public Attributes | |
| NormType | norm = NormType::LayerNorm |
| float | norm_eps = 1e-5f |
The norms' stabilizer (Hugging Face layer_norm_eps; ESM-2 uses 1e-5). | |
| NormPosition | norm_position = NormPosition::Pre |
| MlpType | mlp = MlpType::Plain |
| ElementwiseOp | activation = ElementwiseOp::Gelu |
| The plain MLP's activation (Gelu for BERT and ESM). Ignored for a gated MLP. | |
| GatedActivation | gate_activation = GatedActivation::Silu |
| The gated MLP's gate activation. Ignored for a plain MLP. | |
| bool | mlp_bias = true |
| Biases on the MLP's projections. | |
Everything about an EncoderBlock beyond its attention. The defaults are ESM-2's.
| ElementwiseOp pulsatrix::EncoderBlockOptions::activation = ElementwiseOp::Gelu |
The plain MLP's activation (Gelu for BERT and ESM). Ignored for a gated MLP.
| GatedActivation pulsatrix::EncoderBlockOptions::gate_activation = GatedActivation::Silu |
The gated MLP's gate activation. Ignored for a plain MLP.
| MlpType pulsatrix::EncoderBlockOptions::mlp = MlpType::Plain |
| bool pulsatrix::EncoderBlockOptions::mlp_bias = true |
Biases on the MLP's projections.
| NormType pulsatrix::EncoderBlockOptions::norm = NormType::LayerNorm |
| float pulsatrix::EncoderBlockOptions::norm_eps = 1e-5f |
The norms' stabilizer (Hugging Face layer_norm_eps; ESM-2 uses 1e-5).
| NormPosition pulsatrix::EncoderBlockOptions::norm_position = NormPosition::Pre |