|
pulsatrix 1.1.0
|
A configurable transformer encoder layer (PLM-1): LayerNorm or RMSNorm, before or after each sublayer, and a plain or gated MLP. ESM-2's layer, BERT's layer and the Llama-style pre-norm layer are all settings of it. More...
#include <memory>#include <optional>#include "pulsatrix/mlp_hook.hpp"#include "pulsatrix/module.hpp"#include "pulsatrix/multihead_attention_module.hpp"#include "pulsatrix/swiglu_module.hpp"

Go to the source code of this file.
Classes | |
| struct | pulsatrix::EncoderBlockOptions |
| Everything about an EncoderBlock beyond its attention. The defaults are ESM-2's. More... | |
| class | pulsatrix::EncoderBlock |
One encoder layer. Shape (N, L, d_model) -> (N, L, d_model). More... | |
Namespaces | |
| namespace | pulsatrix |
Enumerations | |
| enum class | pulsatrix::NormType { pulsatrix::LayerNorm , pulsatrix::RMSNorm } |
| The layer's normalization. More... | |
| enum class | pulsatrix::NormPosition { pulsatrix::Pre , pulsatrix::Post } |
| Where the norms sit. More... | |
| enum class | pulsatrix::MlpType { pulsatrix::Plain , pulsatrix::Gated } |
| The MLP's form. More... | |
A configurable transformer encoder layer (PLM-1): LayerNorm or RMSNorm, before or after each sublayer, and a plain or gated MLP. ESM-2's layer, BERT's layer and the Llama-style pre-norm layer are all settings of it.