pulsatrix 1.1.0
Loading...
Searching...
No Matches
encoder_block.hpp File Reference

A configurable transformer encoder layer (PLM-1): LayerNorm or RMSNorm, before or after each sublayer, and a plain or gated MLP. ESM-2's layer, BERT's layer and the Llama-style pre-norm layer are all settings of it. More...

#include <memory>
#include <optional>
#include "pulsatrix/mlp_hook.hpp"
#include "pulsatrix/module.hpp"
#include "pulsatrix/multihead_attention_module.hpp"
#include "pulsatrix/swiglu_module.hpp"
Include dependency graph for encoder_block.hpp:
This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes

struct  pulsatrix::EncoderBlockOptions
 Everything about an EncoderBlock beyond its attention. The defaults are ESM-2's. More...
 
class  pulsatrix::EncoderBlock
 One encoder layer. Shape (N, L, d_model) -> (N, L, d_model). More...
 

Namespaces

namespace  pulsatrix
 

Enumerations

enum class  pulsatrix::NormType { pulsatrix::LayerNorm , pulsatrix::RMSNorm }
 The layer's normalization. More...
 
enum class  pulsatrix::NormPosition { pulsatrix::Pre , pulsatrix::Post }
 Where the norms sit. More...
 
enum class  pulsatrix::MlpType { pulsatrix::Plain , pulsatrix::Gated }
 The MLP's form. More...
 

Detailed Description

A configurable transformer encoder layer (PLM-1): LayerNorm or RMSNorm, before or after each sublayer, and a plain or gated MLP. ESM-2's layer, BERT's layer and the Llama-style pre-norm layer are all settings of it.