safetensors.hpp File Reference

Native safetensors reader and writer (roadmap IO-1): the one file format pulsatrix reads and writes for weights. More...

#include <cstddef>
#include <cstdint>
#include <map>
#include <string>
#include <utility>
#include <vector>
#include "pulsatrix/tensor.hpp"

Include dependency graph for safetensors.hpp:

Go to the source code of this file.

Classes
struct	pulsatrix::SafetensorsTensorInfo
	One tensor's header entry. Offsets are relative to the start of the data section. More...
class	pulsatrix::SafetensorsFile
	A parsed, fully validated safetensors file held in memory. More...

Namespaces
namespace	pulsatrix

Enumerations
enum class	pulsatrix::SafetensorsDtype {   pulsatrix::Bool , pulsatrix::U8 , pulsatrix::I8 , pulsatrix::I16 ,   pulsatrix::U16 , pulsatrix::I32 , pulsatrix::U32 , pulsatrix::I64 ,   pulsatrix::U64 , pulsatrix::F8_E4M3 , pulsatrix::F8_E5M2 , pulsatrix::F16 ,   pulsatrix::BF16 , pulsatrix::F32 , pulsatrix::F64 }
	Element types a safetensors file can declare. Only F32 converts to a Tensor so far. More...

Functions
std::vector< uint8_t >	pulsatrix::SerializeSafetensors (const std::vector< std::pair< std::string, const Tensor * > > &tensors, const std::map< std::string, std::string > &metadata={})
	Serializes tensors (as F32) and string metadata into safetensors bytes.
void	pulsatrix::WriteSafetensors (const std::string &path, const std::vector< std::pair< std::string, const Tensor * > > &tensors, const std::map< std::string, std::string > &metadata={})
	SerializeSafetensors() written to path.

Detailed Description

Native safetensors reader and writer (roadmap IO-1): the one file format pulsatrix reads and writes for weights.

Note

A safetensors file is an 8-byte little-endian header length N, N bytes of JSON, then a packed little-endian data section. It holds only numbers, so loading one can't run code, unlike a pickle (.pt, .pth, .pkl).

The reader treats every file as untrusted. It accepts only what the format allows and throws std::invalid_argument for anything else: a header longer than the file or than 100 MB, JSON outside the format's subset (duplicate keys, unknown fields, non-integer or negative dimensions, invalid UTF-8), a byte range past the end of the data, a range whose size isn't element count times element size (computed without overflow), and ranges that overlap, leave holes, or leave bytes at the end unindexed.

Two deliberate differences from the reference implementation (huggingface/safetensors 0.8), which accepts both: a tensor name that appears twice is rejected (the reference keeps the last, so two tools can disagree about a file's weights), and so is an unknown field in a tensor entry (the format defines exactly dtype, shape and data_offsets). Every file the reference writes passes.