|
pulsatrix 1.1.0
|
Everything that shapes a MultiHeadAttentionModule (LLM-1). More...
#include <multihead_attention_module.hpp>
Public Attributes | |
| int64_t | d_model = 0 |
| int64_t | num_heads = 0 |
| int64_t | num_kv_heads = 0 |
| int64_t | head_dim = 0 |
| bool | use_rope = true |
| RoPELayout | rope_layout = RoPELayout::AdjacentPairs |
| float | rope_base = 10000.0f |
| std::vector< double > | rope_inverse_frequencies |
Explicit RoPE frequencies, one per pair (head_dim / 2), overriding rope_base: how scaled RoPE (Llama 3's rope_scaling) is passed in. Empty uses rope_base. | |
| bool | use_qk_norm = false |
| float | qk_norm_eps = 1e-6f |
| bool | qkv_bias = true |
| bool | out_bias = true |
| bool | causal = false |
| int64_t | sliding_window = 0 |
| float | score_scale = 0.0f |
| float | qk_norm_weight_offset = 0.0f |
Everything that shapes a MultiHeadAttentionModule (LLM-1).
The defaults reproduce the original constructor: as many K/V heads as query heads, head_dim = d_model / num_heads, biases on all four projections, adjacent-pair RoPE with base 10000, no mask. A Llama-family checkpoint (SmolLM2, Llama 3.2) sets num_kv_heads, rope_layout = RoPELayout::RotateHalf, rope_base (its rope_theta), qkv_bias = false, out_bias = false and causal = true. Qwen2.5 is the same with qkv_bias = true; Qwen3 adds use_qk_norm and an explicit head_dim.
| bool pulsatrix::AttentionConfig::causal = false |
Query i attends only to keys 0..i (next-token language models).
| int64_t pulsatrix::AttentionConfig::d_model = 0 |
| int64_t pulsatrix::AttentionConfig::head_dim = 0 |
Per-head width. 0 means d_model / num_heads. Qwen3 and Gemma 3 set it explicitly, so num_heads * head_dim need not equal d_model.
| int64_t pulsatrix::AttentionConfig::num_heads = 0 |
| int64_t pulsatrix::AttentionConfig::num_kv_heads = 0 |
K/V heads, shared by groups of num_heads / num_kv_heads query heads (grouped-query attention; 1 is multi-query attention). 0 means num_heads.
| bool pulsatrix::AttentionConfig::out_bias = true |
Bias on the output projection.
| float pulsatrix::AttentionConfig::qk_norm_eps = 1e-6f |
QK-Norm's RMSNorm stabilizer (Hugging Face rms_norm_eps).
| float pulsatrix::AttentionConfig::qk_norm_weight_offset = 0.0f |
QK-Norm scales by offset + gamma (Gemma's 1 + w; LLM-9).
| bool pulsatrix::AttentionConfig::qkv_bias = true |
Biases on the Q, K and V projections.
| float pulsatrix::AttentionConfig::rope_base = 10000.0f |
| std::vector<double> pulsatrix::AttentionConfig::rope_inverse_frequencies |
Explicit RoPE frequencies, one per pair (head_dim / 2), overriding rope_base: how scaled RoPE (Llama 3's rope_scaling) is passed in. Empty uses rope_base.
| RoPELayout pulsatrix::AttentionConfig::rope_layout = RoPELayout::AdjacentPairs |
| float pulsatrix::AttentionConfig::score_scale = 0.0f |
The factor scores are multiplied by before the softmax. 0 means 1 / sqrt(head_dim); Gemma uses query_pre_attn_scalar^-0.5 (LLM-9).
| int64_t pulsatrix::AttentionConfig::sliding_window = 0 |
With causal, query i sees only its last sliding_window positions, itself included (Hugging Face's sliding-window layers, as in Gemma 3 and Mistral; LLM-9). 0 is no window.
| bool pulsatrix::AttentionConfig::use_qk_norm = false |
| bool pulsatrix::AttentionConfig::use_rope = true |