pulsatrix 1.1.0
Loading...
Searching...
No Matches
benchmark.hpp File Reference

Timing, benchmark reports (pulsatrix.benchmark.v1) and regression comparison. More...

#include <cstdint>
#include <functional>
#include <map>
#include <string>
#include <string_view>
#include <vector>
Include dependency graph for benchmark.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::TimingStats
 Summary of repeated timings, in milliseconds. More...
 
struct  pulsatrix::BenchmarkRecord
 One measured quantity. More...
 
struct  pulsatrix::BenchmarkReport
 A run of benchmarks: pulsatrix.benchmark.v1. More...
 
struct  pulsatrix::CompareOptions
 How CompareBenchmarks decides that a candidate got worse. More...
 
struct  pulsatrix::BenchmarkComparison
 One record compared across two builds. More...
 

Namespaces

namespace  pulsatrix
 

Functions

TimingStats pulsatrix::SummarizeTimings (std::vector< double > samples_ms)
 Summarizes timings. Percentiles interpolate linearly between the sorted samples.
 
TimingStats pulsatrix::TimeIt (const std::function< void()> &fn, int warmup, int repeats)
 Runs fn warmup times untimed, then repeats times timed with a steady clock.
 
std::string pulsatrix::ToJson (const BenchmarkReport &report)
 
BenchmarkReport pulsatrix::ParseBenchmarkReport (std::string_view json)
 Reads a report, following the viz document rules (docs/visualization): schema and version checks, unknown fields ignored, "nonfinite" for NaN and infinity, and errors that name the JSON Pointer of the problem.
 
std::vector< BenchmarkComparison > pulsatrix::CompareBenchmarks (const std::vector< BenchmarkReport > &baseline_runs, const std::vector< BenchmarkReport > &candidate_runs, const CompareOptions &options={})
 Compares a candidate build with a baseline, record by record (matched on name and device). Several reports per side are the rounds of an interleaved A/B run; each side's value is the median over its rounds, so one noisy round doesn't decide, and with three or more rounds per side a time regresses only if its rounds also separate (CompareOptions::significance).
 

Detailed Description

Timing, benchmark reports (pulsatrix.benchmark.v1) and regression comparison.

The pulsatrix_bench tool (tools/bench/) uses these to measure training step time, explanation time and LRP conservation error on every backend a build has, and to compare two builds. They are public so a project can benchmark its own models the same way.