pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::TransformerBlockOptions Struct Reference

The parts of a TransformerBlock beyond its attention (LLM-9 adds Gemma's). More...

#include <transformer_block.hpp>

Public Attributes

float norm_eps = 1e-6f
 Stabilizer of the RMSNorms (Hugging Face rms_norm_eps).
 
bool mlp_bias = true
 Biases on the MLP projections (Hugging Face mlp_bias).
 
GatedActivation mlp_activation = GatedActivation::Silu
 The MLP's gate activation: SiLU, or GELU-tanh for Gemma.
 
bool post_norms = false
 Gemma's sandwich norms: the attention's and the MLP's outputs are normalized again before each residual add (post_attention_layernorm, post_feedforward_layernorm).
 
float norm_weight_offset = 0.0f
 Every RMSNorm scales by offset + gamma (Gemma's 1 + w).
 

Detailed Description

The parts of a TransformerBlock beyond its attention (LLM-9 adds Gemma's).

Member Data Documentation

◆ mlp_activation

GatedActivation pulsatrix::TransformerBlockOptions::mlp_activation = GatedActivation::Silu

The MLP's gate activation: SiLU, or GELU-tanh for Gemma.

◆ mlp_bias

bool pulsatrix::TransformerBlockOptions::mlp_bias = true

Biases on the MLP projections (Hugging Face mlp_bias).

◆ norm_eps

float pulsatrix::TransformerBlockOptions::norm_eps = 1e-6f

Stabilizer of the RMSNorms (Hugging Face rms_norm_eps).

◆ norm_weight_offset

float pulsatrix::TransformerBlockOptions::norm_weight_offset = 0.0f

Every RMSNorm scales by offset + gamma (Gemma's 1 + w).

◆ post_norms

bool pulsatrix::TransformerBlockOptions::post_norms = false

Gemma's sandwich norms: the attention's and the MLP's outputs are normalized again before each residual add (post_attention_layernorm, post_feedforward_layernorm).


The documentation for this struct was generated from the following file: