pulsatrix 1.1.0
Loading...
Searching...
No Matches
hf_model.hpp
Go to the documentation of this file.
1
5#pragma once
6
7#include <cstdint>
8#include <map>
9#include <memory>
10#include <optional>
11#include <string>
12#include <vector>
13
16
17namespace pulsatrix {
18
22 std::string type;
23 float factor = 1.0f;
24 float low_freq_factor = 1.0f;
25 float high_freq_factor = 1.0f;
27};
28
35 std::string architecture;
37 std::string model_type;
38 int64_t hidden_size = 0;
39 int64_t intermediate_size = 0;
40 int64_t num_hidden_layers = 0;
45 int64_t head_dim = 0;
46 int64_t vocab_size = 0;
47 float rms_norm_eps = 1e-6f;
50 float rope_theta = 10000.0f;
52 std::optional<HfRopeScaling> rope_scaling;
55 std::optional<float> rope_local_base_freq;
59 bool qkv_bias = false;
61 bool out_bias = false;
62 bool mlp_bias = false;
64 bool qk_norm = false;
68 bool causal = true;
70 std::string hidden_act = "silu";
72 std::optional<int64_t> bos_token_id;
74 std::vector<int64_t> eos_token_ids;
76 std::string dtype;
78 std::optional<int64_t> sliding_window;
79 std::optional<float> query_pre_attn_scalar;
81 std::vector<std::string> layer_types;
84 float norm_weight_offset = 0.0f;
86 float embed_scale = 1.0f;
89 bool post_norms = false;
90
93 [[nodiscard]] bool is_sliding_layer(int64_t i) const;
100 std::vector<std::string> unsupported;
101};
102
111[[nodiscard]] HfModelConfig ParseHfConfig(std::string_view json);
112
114[[nodiscard]] HfModelConfig ReadHfConfig(const std::string& path);
115
123[[nodiscard]] std::vector<double> RopeInverseFrequencies(const HfModelConfig& config);
124
129
132[[nodiscard]] AttentionConfig ToAttentionConfig(const HfModelConfig& config, int64_t layer);
133
140public:
148 [[nodiscard]] static HfCheckpoint Open(const std::string& directory);
149
151 [[nodiscard]] const std::vector<std::string>& names() const { return names_; }
152 [[nodiscard]] bool contains(const std::string& name) const { return shard_of_.count(name) != 0; }
154 [[nodiscard]] const std::string& shard_of(const std::string& name) const;
156 [[nodiscard]] const SafetensorsTensorInfo& info(const std::string& name) const;
158 [[nodiscard]] Tensor tensor(const std::string& name, DeviceBackend* backend) const;
160 [[nodiscard]] std::vector<std::string> shards() const;
161
162private:
163 HfCheckpoint() = default;
164 std::map<std::string, SafetensorsFile> files_;
165 std::map<std::string, std::string> shard_of_;
166 std::vector<std::string> names_;
167 std::vector<std::string> shard_order_;
168};
169
170} // namespace pulsatrix
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
A Hugging Face checkpoint directory's weights: model.safetensors, or the shards listed by model....
Definition hf_model.hpp:139
std::vector< std::string > shards() const
The shard files, in the order first listed.
const std::string & shard_of(const std::string &name) const
The shard file holding name.
const SafetensorsTensorInfo & info(const std::string &name) const
name's header entry (dtype, shape).
static HfCheckpoint Open(const std::string &directory)
Opens directory: the index if there is one, otherwise the single file.
Tensor tensor(const std::string &name, DeviceBackend *backend) const
Reads name into a Tensor; see SafetensorsFile::tensor for the dtypes it converts.
bool contains(const std::string &name) const
Definition hf_model.hpp:152
const std::vector< std::string > & names() const
Every tensor name, sorted.
Definition hf_model.hpp:151
N-dimensional tensor. Owns its data buffer exclusively; a DeviceBackend* is injected (not owned) – th...
Definition tensor.hpp:29
Multi-head scaled dot-product attention – this codebase's first Module composed out of other real Mod...
Definition acquisition_functions.hpp:16
AttentionConfig ToAttentionConfig(const HfModelConfig &config)
The attention layer the config describes: rotate-half RoPE (with its scaling), causal unless the conf...
HfModelConfig ParseHfConfig(std::string_view json)
Parses config.json text. A multimodal config's text_config is read for the language model.
std::vector< double > RopeInverseFrequencies(const HfModelConfig &config)
The RoPE frequency of each pair, head_dim / 2 of them, as Hugging Face computes them: rope_theta^(-2i...
HfModelConfig ReadHfConfig(const std::string &path)
ParseHfConfig on a file.
Native safetensors reader and writer (roadmap IO-1): the one file format pulsatrix reads and writes f...
Everything that shapes a MultiHeadAttentionModule (LLM-1).
Definition multihead_attention_module.hpp:31
A decoder-only language model's config.json, with the values Hugging Face leaves to the model class f...
Definition hf_model.hpp:33
float embed_scale
Token embeddings are multiplied by this: sqrt(hidden_size) for Gemma 3.
Definition hf_model.hpp:86
std::string model_type
For example "llama", "qwen2", "qwen3", "gemma3_text".
Definition hf_model.hpp:37
std::optional< int64_t > sliding_window
The attention window when sliding-window attention is actually used.
Definition hf_model.hpp:78
std::optional< int64_t > bos_token_id
Definition hf_model.hpp:72
bool causal
false: every position attends to every other. dLLM's A2D ("autoregressive to diffusion") check...
Definition hf_model.hpp:68
int64_t head_dim
head_dim, or hidden_size / num_attention_heads when absent.
Definition hf_model.hpp:45
std::vector< std::string > layer_types
layer_types, for example "sliding_attention" / "full_attention" per layer.
Definition hf_model.hpp:81
int64_t num_attention_heads
Definition hf_model.hpp:41
std::string dtype
torch_dtype (newer files: dtype), for example "bfloat16".
Definition hf_model.hpp:76
int64_t intermediate_size
Definition hf_model.hpp:39
int64_t hidden_size
Definition hf_model.hpp:38
bool mlp_bias
Definition hf_model.hpp:62
bool qk_norm
RMSNorm on each head's queries and keys (Qwen3, Gemma 3).
Definition hf_model.hpp:64
std::optional< HfRopeScaling > rope_scaling
The full-attention layers' RoPE scaling (all layers' when there is one kind).
Definition hf_model.hpp:52
std::vector< int64_t > eos_token_ids
eos_token_id, which may be one id or a list.
Definition hf_model.hpp:74
std::vector< std::string > unsupported
Everything in the config that pulsatrix can't run yet, each naming the roadmap item that adds it (for...
Definition hf_model.hpp:100
int64_t max_position_embeddings
Definition hf_model.hpp:71
bool is_sliding_layer(int64_t i) const
Whether layer i uses sliding-window attention: per layer_types when given, otherwise every layer when...
float rms_norm_eps
Definition hf_model.hpp:47
int64_t num_hidden_layers
Definition hf_model.hpp:40
bool out_bias
Output projection bias (Llama's attention_bias covers it; Qwen2 never has one).
Definition hf_model.hpp:61
float norm_weight_offset
RMSNorms scale by offset + w: 1 for Gemma 3, whose norm weights are stored centered on zero (LLM-9).
Definition hf_model.hpp:84
int64_t num_key_value_heads
num_key_value_heads, or num_attention_heads when absent.
Definition hf_model.hpp:43
std::optional< float > rope_local_base_freq
Sliding-window layers' own RoPE base, unscaled (Gemma 3's rope_local_base_freq, or rope_parameters....
Definition hf_model.hpp:55
std::string hidden_act
hidden_act (Gemma: hidden_activation), for example "silu".
Definition hf_model.hpp:70
std::string architecture
architectures[0], for example "LlamaForCausalLM".
Definition hf_model.hpp:35
bool tie_word_embeddings
Definition hf_model.hpp:56
bool qkv_bias
Q, K and V projection biases: attention_bias, or the architecture's own rule (Qwen2 always has them).
Definition hf_model.hpp:59
float rope_theta
rope_theta (or rope_parameters.rope_theta), 10000 when absent. With per-layer-type RoPE (Gemma 3) it ...
Definition hf_model.hpp:50
std::optional< float > query_pre_attn_scalar
Definition hf_model.hpp:79
bool post_norms
Gemma 3's sandwich norms: attention and MLP outputs are normalized again before each residual add.
Definition hf_model.hpp:89
int64_t vocab_size
Definition hf_model.hpp:46
A rope_scaling (or rope_parameters) entry other than the default.
Definition hf_model.hpp:20
float factor
Definition hf_model.hpp:23
std::string type
rope_type (older files: type), for example "llama3", "linear", "yarn".
Definition hf_model.hpp:22
float low_freq_factor
Definition hf_model.hpp:24
int64_t original_max_position_embeddings
Definition hf_model.hpp:26
float high_freq_factor
Definition hf_model.hpp:25
One tensor's header entry. Offsets are relative to the start of the data section.
Definition safetensors.hpp:38