pulsatrix 1.1.0
Loading...
Searching...
No Matches
golden_logits.hpp
Go to the documentation of this file.
1
6#pragma once
7
8#include <cstdint>
9#include <map>
10#include <string>
11#include <vector>
12
15
16namespace pulsatrix {
17
20 int64_t num_tokens = 0;
22 float max_abs_diff = 0.0f;
23 float mean_abs_diff = 0.0f;
26 float max_scaled_diff = 0.0f;
29};
30
33 std::vector<GoldenSequenceResult> sequences;
35 float max_abs_diff = 0.0f;
37 float max_scaled_diff = 0.0f;
38 float threshold = 0.0f;
40 bool passed = false;
42 std::map<std::string, std::string> metadata;
43};
44
53[[nodiscard]] GoldenSequenceResult CompareLogits(const std::vector<float>& got, const std::vector<float>& expected,
54 int64_t num_tokens, int64_t vocab);
55
64[[nodiscard]] GoldenReport CompareToGolden(CausalLM& model, const SafetensorsFile& golden, float threshold = 1e-3f);
65
66} // namespace pulsatrix
A Llama-family decoder-only language model built from a Hugging Face config, and loading its Hugging ...
Token embedding, num_hidden_layers pre-norm transformer blocks, a final RMSNorm and a language-model ...
Definition causal_lm.hpp:82
A parsed, fully validated safetensors file, held in memory or memory-mapped. Copies share the same by...
Definition safetensors.hpp:47
Definition acquisition_functions.hpp:16
GoldenReport CompareToGolden(CausalLM &model, const SafetensorsFile &golden, float threshold=1e-3f)
Runs model on every ids.<k> sequence of a golden file (as written by tools/golden/make_golden....
GoldenSequenceResult CompareLogits(const std::vector< float > &got, const std::vector< float > &expected, int64_t num_tokens, int64_t vocab)
Compares one sequence's logits, both (L, vocab) row-major.
Native safetensors reader and writer (roadmap IO-1): the one file format pulsatrix reads and writes f...
A model's comparison against a golden file.
Definition golden_logits.hpp:32
float max_abs_diff
The largest max_abs_diff over all sequences.
Definition golden_logits.hpp:35
std::map< std::string, std::string > metadata
The golden file's metadata: model, revision, versions.
Definition golden_logits.hpp:42
bool passed
max_scaled_diff < threshold.
Definition golden_logits.hpp:40
float threshold
Definition golden_logits.hpp:38
float max_scaled_diff
The largest max_scaled_diff over all sequences.
Definition golden_logits.hpp:37
std::vector< GoldenSequenceResult > sequences
Definition golden_logits.hpp:33
One reference sequence's comparison.
Definition golden_logits.hpp:19
float max_scaled_diff
The largest |pulsatrix - reference| / max(1, max |reference logit| at that position): what the thresh...
Definition golden_logits.hpp:26
int64_t num_tokens
Definition golden_logits.hpp:20
float mean_abs_diff
Definition golden_logits.hpp:23
float max_abs_diff
The largest |pulsatrix - reference| over every logit of every position.
Definition golden_logits.hpp:22
int64_t argmax_disagreements
Positions whose most likely next token differs from the reference's.
Definition golden_logits.hpp:28