pulsatrix 1.1.0
Loading...
Searching...
No Matches
pulsatrix::CompareOptions Struct Reference

How CompareBenchmarks decides that a candidate got worse. More...

#include <benchmark.hpp>

Public Attributes

double time_tolerance = 0.10
 A time regresses when the candidate's median exceeds the baseline's by more than this fraction.
 
double metric_tolerance = 1e-5
 A metric regresses when the candidate exceeds the baseline by more than this absolute amount (metrics such as conservation error are near zero, so a fraction of the baseline would be meaningless).
 
double significance = 0.05
 With at least three rounds on each side, a time also needs the candidate's rounds to be slower than the baseline's with this one-sided Mann-Whitney p-value. Separate processes of the same build can differ by 20% on sub-millisecond work, so medians alone raise false alarms; a real slowdown makes every round slower.
 

Detailed Description

How CompareBenchmarks decides that a candidate got worse.

Member Data Documentation

◆ metric_tolerance

double pulsatrix::CompareOptions::metric_tolerance = 1e-5

A metric regresses when the candidate exceeds the baseline by more than this absolute amount (metrics such as conservation error are near zero, so a fraction of the baseline would be meaningless).

◆ significance

double pulsatrix::CompareOptions::significance = 0.05

With at least three rounds on each side, a time also needs the candidate's rounds to be slower than the baseline's with this one-sided Mann-Whitney p-value. Separate processes of the same build can differ by 20% on sub-millisecond work, so medians alone raise false alarms; a real slowdown makes every round slower.

◆ time_tolerance

double pulsatrix::CompareOptions::time_tolerance = 0.10

A time regresses when the candidate's median exceeds the baseline's by more than this fraction.


The documentation for this struct was generated from the following file: