pulsatrix 1.1.0
Loading...
Searching...
No Matches
benchmark.hpp
Go to the documentation of this file.
1
9#pragma once
10
11#include <cstdint>
12#include <functional>
13#include <map>
14#include <string>
15#include <string_view>
16#include <vector>
17
18namespace pulsatrix {
19
22 int64_t repeats = 0;
23 double min_ms = 0.0;
24 double p10_ms = 0.0;
25 double median_ms = 0.0;
26 double p90_ms = 0.0;
27 double mean_ms = 0.0;
28};
29
34[[nodiscard]] TimingStats SummarizeTimings(std::vector<double> samples_ms);
35
43[[nodiscard]] TimingStats TimeIt(const std::function<void()>& fn, int warmup, int repeats);
44
47 enum class Kind {
49 Time,
51 Metric,
52 };
54 std::string name;
56 std::string device;
61 double value = 0.0;
63 std::string unit;
64};
65
69 std::string label;
71 std::map<std::string, std::string> environment;
72 std::vector<BenchmarkRecord> records;
73};
74
76[[nodiscard]] std::string ToJson(const BenchmarkReport& report);
80[[nodiscard]] BenchmarkReport ParseBenchmarkReport(std::string_view json);
81
86 double time_tolerance = 0.10;
90 double metric_tolerance = 1e-5;
95 double significance = 0.05;
96};
97
100 std::string name;
101 std::string device;
105 double baseline = 0.0;
106 double candidate = 0.0;
108 double change = 0.0;
109 bool regression = false;
113 double p_value = 0.0;
115 std::string missing_from;
116};
117
129[[nodiscard]] std::vector<BenchmarkComparison> CompareBenchmarks(const std::vector<BenchmarkReport>& baseline_runs,
130 const std::vector<BenchmarkReport>& candidate_runs,
131 const CompareOptions& options = {});
132
133} // namespace pulsatrix
Definition acquisition_functions.hpp:16
JsonValue ToJson(const AttackResult &result, const AttackOptions &options)
The result as a pulsatrix.attack.v1 document, without the adversarial inputs.
TimingStats TimeIt(const std::function< void()> &fn, int warmup, int repeats)
Runs fn warmup times untimed, then repeats times timed with a steady clock.
std::vector< BenchmarkComparison > CompareBenchmarks(const std::vector< BenchmarkReport > &baseline_runs, const std::vector< BenchmarkReport > &candidate_runs, const CompareOptions &options={})
Compares a candidate build with a baseline, record by record (matched on name and device)....
TimingStats SummarizeTimings(std::vector< double > samples_ms)
Summarizes timings. Percentiles interpolate linearly between the sorted samples.
BenchmarkReport ParseBenchmarkReport(std::string_view json)
Reads a report, following the viz document rules (docs/visualization): schema and version checks,...
One record compared across two builds.
Definition benchmark.hpp:99
std::string device
Definition benchmark.hpp:101
BenchmarkRecord::Kind kind
Definition benchmark.hpp:102
double p_value
For times with at least three rounds per side: the one-sided Mann-Whitney U test's p-value that the c...
Definition benchmark.hpp:113
std::string name
Definition benchmark.hpp:100
std::string missing_from
Set when the record is in only one of the two builds' reports.
Definition benchmark.hpp:115
double baseline
The baseline's and candidate's median time (median over runs of each run's median),...
Definition benchmark.hpp:105
double candidate
Definition benchmark.hpp:106
bool regression
Definition benchmark.hpp:109
double change
candidate / baseline - 1 for times; candidate - baseline for metrics.
Definition benchmark.hpp:108
One measured quantity.
Definition benchmark.hpp:46
Kind
Definition benchmark.hpp:47
@ Metric
A single number where lower is better, such as a relative error: value holds it.
@ Time
A duration: stats holds the timings and lower is better.
TimingStats stats
Set for Kind::Time.
Definition benchmark.hpp:59
double value
Set for Kind::Metric. May be NaN or infinite when the measured thing is broken.
Definition benchmark.hpp:61
std::string unit
What value or stats measure, for example "ms per step" or "relative error".
Definition benchmark.hpp:63
std::string name
What was measured, for example "train.mlp" or "conservation.lrp_epsilon.mlp".
Definition benchmark.hpp:54
std::string device
Where it ran: "cpu", "hip" or "cuda".
Definition benchmark.hpp:56
Kind kind
Definition benchmark.hpp:57
A run of benchmarks: pulsatrix.benchmark.v1.
Definition benchmark.hpp:67
std::vector< BenchmarkRecord > records
Definition benchmark.hpp:72
std::map< std::string, std::string > environment
Build and machine facts: version, build type, compiler, device names.
Definition benchmark.hpp:71
std::string label
Free-form name for the run, for example a branch or commit.
Definition benchmark.hpp:69
How CompareBenchmarks decides that a candidate got worse.
Definition benchmark.hpp:83
double significance
With at least three rounds on each side, a time also needs the candidate's rounds to be slower than t...
Definition benchmark.hpp:95
double time_tolerance
A time regresses when the candidate's median exceeds the baseline's by more than this fraction.
Definition benchmark.hpp:86
double metric_tolerance
A metric regresses when the candidate exceeds the baseline by more than this absolute amount (metrics...
Definition benchmark.hpp:90
Summary of repeated timings, in milliseconds.
Definition benchmark.hpp:21
int64_t repeats
Definition benchmark.hpp:22
double p10_ms
Definition benchmark.hpp:24
double min_ms
Definition benchmark.hpp:23
double median_ms
Definition benchmark.hpp:25
double mean_ms
Definition benchmark.hpp:27
double p90_ms
Definition benchmark.hpp:26