pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::AttentionConfig Struct Reference

Everything that shapes a MultiHeadAttentionModule (LLM-1). More...

#include <multihead_attention_module.hpp>

Public Attributes

int64_t d_model = 0
 
int64_t num_heads = 0
 
int64_t num_kv_heads = 0
 
int64_t head_dim = 0
 
bool use_rope = true
 
RoPELayout rope_layout = RoPELayout::AdjacentPairs
 
float rope_base = 10000.0f
 
std::vector< double > rope_inverse_frequencies
 Explicit RoPE frequencies, one per pair (head_dim / 2), overriding rope_base: how scaled RoPE (Llama 3's rope_scaling) is passed in. Empty uses rope_base.
 
bool use_qk_norm = false
 
float qk_norm_eps = 1e-6f
 
bool qkv_bias = true
 
bool out_bias = true
 
bool causal = false
 
int64_t sliding_window = 0
 
float score_scale = 0.0f
 
float qk_norm_weight_offset = 0.0f
 

Detailed Description

Everything that shapes a MultiHeadAttentionModule (LLM-1).

The defaults reproduce the original constructor: as many K/V heads as query heads, head_dim = d_model / num_heads, biases on all four projections, adjacent-pair RoPE with base 10000, no mask. A Llama-family checkpoint (SmolLM2, Llama 3.2) sets num_kv_heads, rope_layout = RoPELayout::RotateHalf, rope_base (its rope_theta), qkv_bias = false, out_bias = false and causal = true. Qwen2.5 is the same with qkv_bias = true; Qwen3 adds use_qk_norm and an explicit head_dim.

Member Data Documentation

◆ causal

bool pulsatrix::AttentionConfig::causal = false

Query i attends only to keys 0..i (next-token language models).

◆ d_model

int64_t pulsatrix::AttentionConfig::d_model = 0

◆ head_dim

int64_t pulsatrix::AttentionConfig::head_dim = 0

Per-head width. 0 means d_model / num_heads. Qwen3 and Gemma 3 set it explicitly, so num_heads * head_dim need not equal d_model.

◆ num_heads

int64_t pulsatrix::AttentionConfig::num_heads = 0

◆ num_kv_heads

int64_t pulsatrix::AttentionConfig::num_kv_heads = 0

K/V heads, shared by groups of num_heads / num_kv_heads query heads (grouped-query attention; 1 is multi-query attention). 0 means num_heads.

◆ out_bias

bool pulsatrix::AttentionConfig::out_bias = true

Bias on the output projection.

◆ qk_norm_eps

float pulsatrix::AttentionConfig::qk_norm_eps = 1e-6f

QK-Norm's RMSNorm stabilizer (Hugging Face rms_norm_eps).

◆ qk_norm_weight_offset

float pulsatrix::AttentionConfig::qk_norm_weight_offset = 0.0f

QK-Norm scales by offset + gamma (Gemma's 1 + w; LLM-9).

◆ qkv_bias

bool pulsatrix::AttentionConfig::qkv_bias = true

Biases on the Q, K and V projections.

◆ rope_base

float pulsatrix::AttentionConfig::rope_base = 10000.0f

◆ rope_inverse_frequencies

std::vector<double> pulsatrix::AttentionConfig::rope_inverse_frequencies

Explicit RoPE frequencies, one per pair (head_dim / 2), overriding rope_base: how scaled RoPE (Llama 3's rope_scaling) is passed in. Empty uses rope_base.

◆ rope_layout

RoPELayout pulsatrix::AttentionConfig::rope_layout = RoPELayout::AdjacentPairs

◆ score_scale

float pulsatrix::AttentionConfig::score_scale = 0.0f

The factor scores are multiplied by before the softmax. 0 means 1 / sqrt(head_dim); Gemma uses query_pre_attn_scalar^-0.5 (LLM-9).

◆ sliding_window

int64_t pulsatrix::AttentionConfig::sliding_window = 0

With causal, query i sees only its last sliding_window positions, itself included (Hugging Face's sliding-window layers, as in Gemma 3 and Mistral; LLM-9). 0 is no window.

◆ use_qk_norm

bool pulsatrix::AttentionConfig::use_qk_norm = false

◆ use_rope

bool pulsatrix::AttentionConfig::use_rope = true

The documentation for this struct was generated from the following file: