pulsatrix 1.1.0
Loading...
Searching...
No Matches
hf_model.hpp File Reference

Hugging Face checkpoints: config.json and single or sharded safetensors (IO-5). More...

#include <cstdint>
#include <map>
#include <memory>
#include <optional>
#include <string>
#include <vector>
#include "pulsatrix/multihead_attention_module.hpp"
#include "pulsatrix/safetensors.hpp"
Include dependency graph for hf_model.hpp:
This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes

struct  pulsatrix::HfRopeScaling
 A rope_scaling (or rope_parameters) entry other than the default. More...
 
struct  pulsatrix::HfModelConfig
 A decoder-only language model's config.json, with the values Hugging Face leaves to the model class filled in: every field is what the model actually uses. More...
 
class  pulsatrix::HfCheckpoint
 A Hugging Face checkpoint directory's weights: model.safetensors, or the shards listed by model.safetensors.index.json (IO-5). Shards are memory-mapped, so opening a checkpoint reads only headers, and a tensor's bytes are paged in when it is read. More...
 

Namespaces

namespace  pulsatrix
 

Functions

HfModelConfig pulsatrix::ParseHfConfig (std::string_view json)
 Parses config.json text. A multimodal config's text_config is read for the language model.
 
HfModelConfig pulsatrix::ReadHfConfig (const std::string &path)
 ParseHfConfig on a file.
 
std::vector< double > pulsatrix::RopeInverseFrequencies (const HfModelConfig &config)
 The RoPE frequency of each pair, head_dim / 2 of them, as Hugging Face computes them: rope_theta^(-2i / head_dim), then rescaled by rope_scaling when it is "linear" (all divided by the factor) or "llama3" (low frequencies divided by the factor, high ones kept, the band between interpolated; LLM-6).
 
AttentionConfig pulsatrix::ToAttentionConfig (const HfModelConfig &config)
 The attention layer the config describes: rotate-half RoPE (with its scaling), causal unless the config says otherwise, its head counts, biases, QK-Norm and epsilon, and Gemma's score scale and QK-Norm offset. For a model whose layers differ, this is a full-attention layer.
 
AttentionConfig pulsatrix::ToAttentionConfig (const HfModelConfig &config, int64_t layer)
 Layer layer's attention (LLM-9): a sliding-window layer gets its window and its own RoPE base, unscaled; a full-attention layer is ToAttentionConfig(config).
 

Detailed Description

Hugging Face checkpoints: config.json and single or sharded safetensors (IO-5).