|
pulsatrix 1.1.0
|
The parts of a TransformerBlock beyond its attention (LLM-9 adds Gemma's). More...
#include <transformer_block.hpp>
Public Attributes | |
| float | norm_eps = 1e-6f |
Stabilizer of the RMSNorms (Hugging Face rms_norm_eps). | |
| bool | mlp_bias = true |
Biases on the MLP projections (Hugging Face mlp_bias). | |
| GatedActivation | mlp_activation = GatedActivation::Silu |
| The MLP's gate activation: SiLU, or GELU-tanh for Gemma. | |
| bool | post_norms = false |
Gemma's sandwich norms: the attention's and the MLP's outputs are normalized again before each residual add (post_attention_layernorm, post_feedforward_layernorm). | |
| float | norm_weight_offset = 0.0f |
Every RMSNorm scales by offset + gamma (Gemma's 1 + w). | |
The parts of a TransformerBlock beyond its attention (LLM-9 adds Gemma's).
| GatedActivation pulsatrix::TransformerBlockOptions::mlp_activation = GatedActivation::Silu |
The MLP's gate activation: SiLU, or GELU-tanh for Gemma.
| bool pulsatrix::TransformerBlockOptions::mlp_bias = true |
Biases on the MLP projections (Hugging Face mlp_bias).
| float pulsatrix::TransformerBlockOptions::norm_eps = 1e-6f |
Stabilizer of the RMSNorms (Hugging Face rms_norm_eps).
| float pulsatrix::TransformerBlockOptions::norm_weight_offset = 0.0f |
Every RMSNorm scales by offset + gamma (Gemma's 1 + w).
| bool pulsatrix::TransformerBlockOptions::post_norms = false |
Gemma's sandwich norms: the attention's and the MLP's outputs are normalized again before each residual add (post_attention_layernorm, post_feedforward_layernorm).