|
pulsatrix 1.1.0
|
A decoder-only language model's config.json, with the values Hugging Face leaves to the model class filled in: every field is what the model actually uses.
More...
#include <hf_model.hpp>
Public Member Functions | |
| bool | is_sliding_layer (int64_t i) const |
Whether layer i uses sliding-window attention: per layer_types when given, otherwise every layer when sliding_window is set (Mistral). | |
Public Attributes | |
| std::string | architecture |
architectures[0], for example "LlamaForCausalLM". | |
| std::string | model_type |
For example "llama", "qwen2", "qwen3", "gemma3_text". | |
| int64_t | hidden_size = 0 |
| int64_t | intermediate_size = 0 |
| int64_t | num_hidden_layers = 0 |
| int64_t | num_attention_heads = 0 |
| int64_t | num_key_value_heads = 0 |
num_key_value_heads, or num_attention_heads when absent. | |
| int64_t | head_dim = 0 |
head_dim, or hidden_size / num_attention_heads when absent. | |
| int64_t | vocab_size = 0 |
| float | rms_norm_eps = 1e-6f |
| float | rope_theta = 10000.0f |
rope_theta (or rope_parameters.rope_theta), 10000 when absent. With per-layer-type RoPE (Gemma 3) it is the full-attention layers' base. | |
| std::optional< HfRopeScaling > | rope_scaling |
| The full-attention layers' RoPE scaling (all layers' when there is one kind). | |
| std::optional< float > | rope_local_base_freq |
Sliding-window layers' own RoPE base, unscaled (Gemma 3's rope_local_base_freq, or rope_parameters.sliding_attention.rope_theta). Absent: they use rope_theta. | |
| bool | tie_word_embeddings = true |
| bool | qkv_bias = false |
Q, K and V projection biases: attention_bias, or the architecture's own rule (Qwen2 always has them). | |
| bool | out_bias = false |
Output projection bias (Llama's attention_bias covers it; Qwen2 never has one). | |
| bool | mlp_bias = false |
| bool | qk_norm = false |
| RMSNorm on each head's queries and keys (Qwen3, Gemma 3). | |
| bool | causal = true |
false: every position attends to every other. dLLM's A2D ("autoregressive to
diffusion") checkpoints, model_type a2d-qwen3, are Qwen3 with this one change (DIFF-4). Such a model is a masked diffusion denoiser, not a next-token predictor. | |
| std::string | hidden_act = "silu" |
hidden_act (Gemma: hidden_activation), for example "silu". | |
| int64_t | max_position_embeddings = 0 |
| std::optional< int64_t > | bos_token_id |
| std::vector< int64_t > | eos_token_ids |
eos_token_id, which may be one id or a list. | |
| std::string | dtype |
torch_dtype (newer files: dtype), for example "bfloat16". | |
| std::optional< int64_t > | sliding_window |
| The attention window when sliding-window attention is actually used. | |
| std::optional< float > | query_pre_attn_scalar |
| std::vector< std::string > | layer_types |
layer_types, for example "sliding_attention" / "full_attention" per layer. | |
| float | norm_weight_offset = 0.0f |
RMSNorms scale by offset + w: 1 for Gemma 3, whose norm weights are stored centered on zero (LLM-9). | |
| float | embed_scale = 1.0f |
Token embeddings are multiplied by this: sqrt(hidden_size) for Gemma 3. | |
| bool | post_norms = false |
| Gemma 3's sandwich norms: attention and MLP outputs are normalized again before each residual add. | |
| std::vector< std::string > | unsupported |
Everything in the config that pulsatrix can't run yet, each naming the roadmap item that adds it (for example "rope_scaling \"llama3\" (LLM-6)"), including an architecture that isn't a ...ForCausalLM decoder. Empty means the model maps onto pulsatrix's modules exactly; a loader should refuse otherwise. | |
A decoder-only language model's config.json, with the values Hugging Face leaves to the model class filled in: every field is what the model actually uses.
| bool pulsatrix::HfModelConfig::is_sliding_layer | ( | int64_t | i | ) | const |
Whether layer i uses sliding-window attention: per layer_types when given, otherwise every layer when sliding_window is set (Mistral).
| std::string pulsatrix::HfModelConfig::architecture |
architectures[0], for example "LlamaForCausalLM".
| std::optional<int64_t> pulsatrix::HfModelConfig::bos_token_id |
| bool pulsatrix::HfModelConfig::causal = true |
false: every position attends to every other. dLLM's A2D ("autoregressive to
diffusion") checkpoints, model_type a2d-qwen3, are Qwen3 with this one change (DIFF-4). Such a model is a masked diffusion denoiser, not a next-token predictor.
| std::string pulsatrix::HfModelConfig::dtype |
torch_dtype (newer files: dtype), for example "bfloat16".
| float pulsatrix::HfModelConfig::embed_scale = 1.0f |
Token embeddings are multiplied by this: sqrt(hidden_size) for Gemma 3.
| std::vector<int64_t> pulsatrix::HfModelConfig::eos_token_ids |
eos_token_id, which may be one id or a list.
| int64_t pulsatrix::HfModelConfig::head_dim = 0 |
head_dim, or hidden_size / num_attention_heads when absent.
| std::string pulsatrix::HfModelConfig::hidden_act = "silu" |
hidden_act (Gemma: hidden_activation), for example "silu".
| int64_t pulsatrix::HfModelConfig::hidden_size = 0 |
| int64_t pulsatrix::HfModelConfig::intermediate_size = 0 |
| std::vector<std::string> pulsatrix::HfModelConfig::layer_types |
layer_types, for example "sliding_attention" / "full_attention" per layer.
| int64_t pulsatrix::HfModelConfig::max_position_embeddings = 0 |
| bool pulsatrix::HfModelConfig::mlp_bias = false |
| std::string pulsatrix::HfModelConfig::model_type |
For example "llama", "qwen2", "qwen3", "gemma3_text".
| float pulsatrix::HfModelConfig::norm_weight_offset = 0.0f |
RMSNorms scale by offset + w: 1 for Gemma 3, whose norm weights are stored centered on zero (LLM-9).
| int64_t pulsatrix::HfModelConfig::num_attention_heads = 0 |
| int64_t pulsatrix::HfModelConfig::num_hidden_layers = 0 |
| int64_t pulsatrix::HfModelConfig::num_key_value_heads = 0 |
num_key_value_heads, or num_attention_heads when absent.
| bool pulsatrix::HfModelConfig::out_bias = false |
Output projection bias (Llama's attention_bias covers it; Qwen2 never has one).
| bool pulsatrix::HfModelConfig::post_norms = false |
Gemma 3's sandwich norms: attention and MLP outputs are normalized again before each residual add.
| bool pulsatrix::HfModelConfig::qk_norm = false |
RMSNorm on each head's queries and keys (Qwen3, Gemma 3).
| bool pulsatrix::HfModelConfig::qkv_bias = false |
Q, K and V projection biases: attention_bias, or the architecture's own rule (Qwen2 always has them).
| std::optional<float> pulsatrix::HfModelConfig::query_pre_attn_scalar |
| float pulsatrix::HfModelConfig::rms_norm_eps = 1e-6f |
| std::optional<float> pulsatrix::HfModelConfig::rope_local_base_freq |
Sliding-window layers' own RoPE base, unscaled (Gemma 3's rope_local_base_freq, or rope_parameters.sliding_attention.rope_theta). Absent: they use rope_theta.
| std::optional<HfRopeScaling> pulsatrix::HfModelConfig::rope_scaling |
The full-attention layers' RoPE scaling (all layers' when there is one kind).
| float pulsatrix::HfModelConfig::rope_theta = 10000.0f |
rope_theta (or rope_parameters.rope_theta), 10000 when absent. With per-layer-type RoPE (Gemma 3) it is the full-attention layers' base.
| std::optional<int64_t> pulsatrix::HfModelConfig::sliding_window |
The attention window when sliding-window attention is actually used.
| bool pulsatrix::HfModelConfig::tie_word_embeddings = true |
| std::vector<std::string> pulsatrix::HfModelConfig::unsupported |
Everything in the config that pulsatrix can't run yet, each naming the roadmap item that adds it (for example "rope_scaling \"llama3\" (LLM-6)"), including an architecture that isn't a ...ForCausalLM decoder. Empty means the model maps onto pulsatrix's modules exactly; a loader should refuse otherwise.
| int64_t pulsatrix::HfModelConfig::vocab_size = 0 |