pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::HfModelConfig Struct Reference

A decoder-only language model's config.json, with the values Hugging Face leaves to the model class filled in: every field is what the model actually uses. More...

#include <hf_model.hpp>

Public Member Functions

bool is_sliding_layer (int64_t i) const
 Whether layer i uses sliding-window attention: per layer_types when given, otherwise every layer when sliding_window is set (Mistral).
 

Public Attributes

std::string architecture
 architectures[0], for example "LlamaForCausalLM".
 
std::string model_type
 For example "llama", "qwen2", "qwen3", "gemma3_text".
 
int64_t hidden_size = 0
 
int64_t intermediate_size = 0
 
int64_t num_hidden_layers = 0
 
int64_t num_attention_heads = 0
 
int64_t num_key_value_heads = 0
 num_key_value_heads, or num_attention_heads when absent.
 
int64_t head_dim = 0
 head_dim, or hidden_size / num_attention_heads when absent.
 
int64_t vocab_size = 0
 
float rms_norm_eps = 1e-6f
 
float rope_theta = 10000.0f
 rope_theta (or rope_parameters.rope_theta), 10000 when absent. With per-layer-type RoPE (Gemma 3) it is the full-attention layers' base.
 
std::optional< HfRopeScaling > rope_scaling
 The full-attention layers' RoPE scaling (all layers' when there is one kind).
 
std::optional< float > rope_local_base_freq
 Sliding-window layers' own RoPE base, unscaled (Gemma 3's rope_local_base_freq, or rope_parameters.sliding_attention.rope_theta). Absent: they use rope_theta.
 
bool tie_word_embeddings = true
 
bool qkv_bias = false
 Q, K and V projection biases: attention_bias, or the architecture's own rule (Qwen2 always has them).
 
bool out_bias = false
 Output projection bias (Llama's attention_bias covers it; Qwen2 never has one).
 
bool mlp_bias = false
 
bool qk_norm = false
 RMSNorm on each head's queries and keys (Qwen3, Gemma 3).
 
bool causal = true
 false: every position attends to every other. dLLM's A2D ("autoregressive to diffusion") checkpoints, model_type a2d-qwen3, are Qwen3 with this one change (DIFF-4). Such a model is a masked diffusion denoiser, not a next-token predictor.
 
std::string hidden_act = "silu"
 hidden_act (Gemma: hidden_activation), for example "silu".
 
int64_t max_position_embeddings = 0
 
std::optional< int64_t > bos_token_id
 
std::vector< int64_t > eos_token_ids
 eos_token_id, which may be one id or a list.
 
std::string dtype
 torch_dtype (newer files: dtype), for example "bfloat16".
 
std::optional< int64_t > sliding_window
 The attention window when sliding-window attention is actually used.
 
std::optional< float > query_pre_attn_scalar
 
std::vector< std::string > layer_types
 layer_types, for example "sliding_attention" / "full_attention" per layer.
 
float norm_weight_offset = 0.0f
 RMSNorms scale by offset + w: 1 for Gemma 3, whose norm weights are stored centered on zero (LLM-9).
 
float embed_scale = 1.0f
 Token embeddings are multiplied by this: sqrt(hidden_size) for Gemma 3.
 
bool post_norms = false
 Gemma 3's sandwich norms: attention and MLP outputs are normalized again before each residual add.
 
std::vector< std::string > unsupported
 Everything in the config that pulsatrix can't run yet, each naming the roadmap item that adds it (for example "rope_scaling \"llama3\" (LLM-6)"), including an architecture that isn't a ...ForCausalLM decoder. Empty means the model maps onto pulsatrix's modules exactly; a loader should refuse otherwise.
 

Detailed Description

A decoder-only language model's config.json, with the values Hugging Face leaves to the model class filled in: every field is what the model actually uses.

Member Function Documentation

◆ is_sliding_layer()

bool pulsatrix::HfModelConfig::is_sliding_layer ( int64_t  i) const

Whether layer i uses sliding-window attention: per layer_types when given, otherwise every layer when sliding_window is set (Mistral).

Member Data Documentation

◆ architecture

std::string pulsatrix::HfModelConfig::architecture

architectures[0], for example "LlamaForCausalLM".

◆ bos_token_id

std::optional<int64_t> pulsatrix::HfModelConfig::bos_token_id

◆ causal

bool pulsatrix::HfModelConfig::causal = true

false: every position attends to every other. dLLM's A2D ("autoregressive to diffusion") checkpoints, model_type a2d-qwen3, are Qwen3 with this one change (DIFF-4). Such a model is a masked diffusion denoiser, not a next-token predictor.

◆ dtype

std::string pulsatrix::HfModelConfig::dtype

torch_dtype (newer files: dtype), for example "bfloat16".

◆ embed_scale

float pulsatrix::HfModelConfig::embed_scale = 1.0f

Token embeddings are multiplied by this: sqrt(hidden_size) for Gemma 3.

◆ eos_token_ids

std::vector<int64_t> pulsatrix::HfModelConfig::eos_token_ids

eos_token_id, which may be one id or a list.

◆ head_dim

int64_t pulsatrix::HfModelConfig::head_dim = 0

head_dim, or hidden_size / num_attention_heads when absent.

◆ hidden_act

std::string pulsatrix::HfModelConfig::hidden_act = "silu"

hidden_act (Gemma: hidden_activation), for example "silu".

◆ hidden_size

int64_t pulsatrix::HfModelConfig::hidden_size = 0

◆ intermediate_size

int64_t pulsatrix::HfModelConfig::intermediate_size = 0

◆ layer_types

std::vector<std::string> pulsatrix::HfModelConfig::layer_types

layer_types, for example "sliding_attention" / "full_attention" per layer.

◆ max_position_embeddings

int64_t pulsatrix::HfModelConfig::max_position_embeddings = 0

◆ mlp_bias

bool pulsatrix::HfModelConfig::mlp_bias = false

◆ model_type

std::string pulsatrix::HfModelConfig::model_type

For example "llama", "qwen2", "qwen3", "gemma3_text".

◆ norm_weight_offset

float pulsatrix::HfModelConfig::norm_weight_offset = 0.0f

RMSNorms scale by offset + w: 1 for Gemma 3, whose norm weights are stored centered on zero (LLM-9).

◆ num_attention_heads

int64_t pulsatrix::HfModelConfig::num_attention_heads = 0

◆ num_hidden_layers

int64_t pulsatrix::HfModelConfig::num_hidden_layers = 0

◆ num_key_value_heads

int64_t pulsatrix::HfModelConfig::num_key_value_heads = 0

num_key_value_heads, or num_attention_heads when absent.

◆ out_bias

bool pulsatrix::HfModelConfig::out_bias = false

Output projection bias (Llama's attention_bias covers it; Qwen2 never has one).

◆ post_norms

bool pulsatrix::HfModelConfig::post_norms = false

Gemma 3's sandwich norms: attention and MLP outputs are normalized again before each residual add.

◆ qk_norm

bool pulsatrix::HfModelConfig::qk_norm = false

RMSNorm on each head's queries and keys (Qwen3, Gemma 3).

◆ qkv_bias

bool pulsatrix::HfModelConfig::qkv_bias = false

Q, K and V projection biases: attention_bias, or the architecture's own rule (Qwen2 always has them).

◆ query_pre_attn_scalar

std::optional<float> pulsatrix::HfModelConfig::query_pre_attn_scalar

◆ rms_norm_eps

float pulsatrix::HfModelConfig::rms_norm_eps = 1e-6f

◆ rope_local_base_freq

std::optional<float> pulsatrix::HfModelConfig::rope_local_base_freq

Sliding-window layers' own RoPE base, unscaled (Gemma 3's rope_local_base_freq, or rope_parameters.sliding_attention.rope_theta). Absent: they use rope_theta.

◆ rope_scaling

std::optional<HfRopeScaling> pulsatrix::HfModelConfig::rope_scaling

The full-attention layers' RoPE scaling (all layers' when there is one kind).

◆ rope_theta

float pulsatrix::HfModelConfig::rope_theta = 10000.0f

rope_theta (or rope_parameters.rope_theta), 10000 when absent. With per-layer-type RoPE (Gemma 3) it is the full-attention layers' base.

◆ sliding_window

std::optional<int64_t> pulsatrix::HfModelConfig::sliding_window

The attention window when sliding-window attention is actually used.

◆ tie_word_embeddings

bool pulsatrix::HfModelConfig::tie_word_embeddings = true

◆ unsupported

std::vector<std::string> pulsatrix::HfModelConfig::unsupported

Everything in the config that pulsatrix can't run yet, each naming the roadmap item that adds it (for example "rope_scaling \"llama3\" (LLM-6)"), including an architecture that isn't a ...ForCausalLM decoder. Empty means the model maps onto pulsatrix's modules exactly; a loader should refuse otherwise.

◆ vocab_size

int64_t pulsatrix::HfModelConfig::vocab_size = 0

The documentation for this struct was generated from the following file: