pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::EncoderBlockOptions Struct Reference

Everything about an EncoderBlock beyond its attention. The defaults are ESM-2's. More...

#include <encoder_block.hpp>

Public Attributes

NormType norm = NormType::LayerNorm
 
float norm_eps = 1e-5f
 The norms' stabilizer (Hugging Face layer_norm_eps; ESM-2 uses 1e-5).
 
NormPosition norm_position = NormPosition::Pre
 
MlpType mlp = MlpType::Plain
 
ElementwiseOp activation = ElementwiseOp::Gelu
 The plain MLP's activation (Gelu for BERT and ESM). Ignored for a gated MLP.
 
GatedActivation gate_activation = GatedActivation::Silu
 The gated MLP's gate activation. Ignored for a plain MLP.
 
bool mlp_bias = true
 Biases on the MLP's projections.
 

Detailed Description

Everything about an EncoderBlock beyond its attention. The defaults are ESM-2's.

Member Data Documentation

◆ activation

ElementwiseOp pulsatrix::EncoderBlockOptions::activation = ElementwiseOp::Gelu

The plain MLP's activation (Gelu for BERT and ESM). Ignored for a gated MLP.

◆ gate_activation

GatedActivation pulsatrix::EncoderBlockOptions::gate_activation = GatedActivation::Silu

The gated MLP's gate activation. Ignored for a plain MLP.

◆ mlp

MlpType pulsatrix::EncoderBlockOptions::mlp = MlpType::Plain

◆ mlp_bias

bool pulsatrix::EncoderBlockOptions::mlp_bias = true

Biases on the MLP's projections.

◆ norm

NormType pulsatrix::EncoderBlockOptions::norm = NormType::LayerNorm

◆ norm_eps

float pulsatrix::EncoderBlockOptions::norm_eps = 1e-5f

The norms' stabilizer (Hugging Face layer_norm_eps; ESM-2 uses 1e-5).

◆ norm_position

NormPosition pulsatrix::EncoderBlockOptions::norm_position = NormPosition::Pre

The documentation for this struct was generated from the following file: