129[[nodiscard]] std::vector<BenchmarkComparison>
CompareBenchmarks(
const std::vector<BenchmarkReport>& baseline_runs,
130 const std::vector<BenchmarkReport>& candidate_runs,
Definition acquisition_functions.hpp:16
JsonValue ToJson(const AttackResult &result, const AttackOptions &options)
The result as a pulsatrix.attack.v1 document, without the adversarial inputs.
TimingStats TimeIt(const std::function< void()> &fn, int warmup, int repeats)
Runs fn warmup times untimed, then repeats times timed with a steady clock.
std::vector< BenchmarkComparison > CompareBenchmarks(const std::vector< BenchmarkReport > &baseline_runs, const std::vector< BenchmarkReport > &candidate_runs, const CompareOptions &options={})
Compares a candidate build with a baseline, record by record (matched on name and device)....
TimingStats SummarizeTimings(std::vector< double > samples_ms)
Summarizes timings. Percentiles interpolate linearly between the sorted samples.
BenchmarkReport ParseBenchmarkReport(std::string_view json)
Reads a report, following the viz document rules (docs/visualization): schema and version checks,...
One record compared across two builds.
Definition benchmark.hpp:99
std::string device
Definition benchmark.hpp:101
BenchmarkRecord::Kind kind
Definition benchmark.hpp:102
double p_value
For times with at least three rounds per side: the one-sided Mann-Whitney U test's p-value that the c...
Definition benchmark.hpp:113
std::string name
Definition benchmark.hpp:100
std::string missing_from
Set when the record is in only one of the two builds' reports.
Definition benchmark.hpp:115
double baseline
The baseline's and candidate's median time (median over runs of each run's median),...
Definition benchmark.hpp:105
double candidate
Definition benchmark.hpp:106
bool regression
Definition benchmark.hpp:109
double change
candidate / baseline - 1 for times; candidate - baseline for metrics.
Definition benchmark.hpp:108
One measured quantity.
Definition benchmark.hpp:46
Kind
Definition benchmark.hpp:47
@ Metric
A single number where lower is better, such as a relative error: value holds it.
@ Time
A duration: stats holds the timings and lower is better.
TimingStats stats
Set for Kind::Time.
Definition benchmark.hpp:59
double value
Set for Kind::Metric. May be NaN or infinite when the measured thing is broken.
Definition benchmark.hpp:61
std::string unit
What value or stats measure, for example "ms per step" or "relative error".
Definition benchmark.hpp:63
std::string name
What was measured, for example "train.mlp" or "conservation.lrp_epsilon.mlp".
Definition benchmark.hpp:54
std::string device
Where it ran: "cpu", "hip" or "cuda".
Definition benchmark.hpp:56
Kind kind
Definition benchmark.hpp:57
A run of benchmarks: pulsatrix.benchmark.v1.
Definition benchmark.hpp:67
std::vector< BenchmarkRecord > records
Definition benchmark.hpp:72
std::map< std::string, std::string > environment
Build and machine facts: version, build type, compiler, device names.
Definition benchmark.hpp:71
std::string label
Free-form name for the run, for example a branch or commit.
Definition benchmark.hpp:69
How CompareBenchmarks decides that a candidate got worse.
Definition benchmark.hpp:83
double significance
With at least three rounds on each side, a time also needs the candidate's rounds to be slower than t...
Definition benchmark.hpp:95
double time_tolerance
A time regresses when the candidate's median exceeds the baseline's by more than this fraction.
Definition benchmark.hpp:86
double metric_tolerance
A metric regresses when the candidate exceeds the baseline by more than this absolute amount (metrics...
Definition benchmark.hpp:90
Summary of repeated timings, in milliseconds.
Definition benchmark.hpp:21
int64_t repeats
Definition benchmark.hpp:22
double p10_ms
Definition benchmark.hpp:24
double min_ms
Definition benchmark.hpp:23
double median_ms
Definition benchmark.hpp:25
double mean_ms
Definition benchmark.hpp:27
double p90_ms
Definition benchmark.hpp:26